diff --git a/README.md b/README.md index 5fd5053..eb158c4 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,7 @@ assignment application for Django, designed to be dropped into an existing site * Public-facing submission form for tickets * Mobile-friendly (work in progress) * Separate view for My Tasks (across lists) +* Batch-import tasks via CSV ## Requirements @@ -45,14 +46,13 @@ All tasks are "created by" the current user and can optionally be "assigned to" django-todo v2 makes use of features only available in Django 2.0. It will not work in previous versions. v2 is only tested against Python 3.x -- no guarantees if running it against older versions. -# Installation +## Installation django-todo is a Django app, not a project site. It needs a site to live in. You can either install it into an existing Django project site, or clone the django-todo [demo site (GTD)](https://github.com/shacker/gtd). If using your own site, be sure you have jQuery and Bootstrap wired up and working. -django-todo pages that require it will insert additional CSS/JavaScript into page heads, -so your project's base templates must include: +django-todo views that require it will insert additional CSS/JavaScript into page heads, so your project's base templates must include: ```jinja {% block extrahead %}{% endblock extrahead %} @@ -100,13 +100,17 @@ django-todo makes use of the Django `messages` system. Make sure you have someth Log in and access `/todo`! +### Customizing Templates + The provided templates are fairly bare-bones, and are meant as starting points only. Unlike previous versions of django-todo, they now ship as Bootstrap examples, but feel free to override them - there is no hard dependency on Bootstrap. To override a template, create a `todo` folder in your project's `templates` dir, then copy the template you want to override from django-todo source and into that dir. +### Filing Public Tickets + If you wish to use the public ticket-filing system, first create the list into which those tickets should be filed, then add its slug to `TODO_DEFAULT_LIST_SLUG` in settings (more on settings below). ## Settings -Optional configuration options: +Optional configuration params, which can be added to your project settings: ```python # Restrict access to ALL todo lists/views to `is_staff` users. @@ -141,6 +145,46 @@ The current django-todo version number is available from the [todo package](http python -c "import todo; print(todo.__version__)" +## Importing Tasks via CSV + +django-todo has the ability to batch-import ("upsert") tasks from a specifically formatted CSV spreadsheet. This ability is provided through both a management command and a web interface. + +**Management Command** + +`./manage.py import_csv -f /path/to/file.csv` + +**Web Importer** + +Link from your navigation to `{url "todo:import_csv"}` + + +### CSV Formatting + +Copy `todo/data/import_example.csv` to another location on your system and edit in a spreadsheet or directly. + +**Do not edit the header row!** + +The first four columns: `'Title', 'Group', 'Task List', 'Created By'` are required -- all others are optional and should work pretty much exactly like manual task entry via the web UI. + +Note: Internally, Tasks are keyed to TaskLists, not to Groups (TaskLists are in Gruops). However, we request the Group in the CSV +because it's possible to have multiple TaskLists with the same name in different groups; i.e. we need it for namespacing and permissions. + + +### Import Rules + +Because data entered via CSV is not going through the same view permissions enforced in the rest of django-todo, and to simplify data dependency logic, and to pre-empt disagreements between django-todo users, the importer will *not* create new users, groups, or task lists. All users, groups, and task lists referenced in your CSV must already exist, and group memberships must be correct. + +Any validation error (e.g. unparse-able dates, incorrect group memberships) **will result in that row being skipped.** + +A report of rows upserted and rows skipped (with line numbers and reasons) is provided at the end of the run. + +### Upsert Logic + +For each valid row, we need to decide whether to create a new task or update an existing one. django-todo matches on the unique combination of the four required columns. If we find a task that matches those, we *update* the rest of the columns. In other words, if you import a CSV once, then edit the Assigned To for a task and import it again, the original task will be updated with a new assignee (and same for the other columns). + +Otherwise we create a new task. + + ## Mail Tracking What if you could turn django-todo into a shared mailbox? Django-todo includes an optional feature that allows emails @@ -208,6 +252,8 @@ A mail worker can be started with: ./manage.py mail_worker test_tracker ``` +Some views and URLs were renamed in 2.0 for logical consistency. If this affects you, see source code and the demo GTD site for reference to the new URL names. + If you want to log mail events, make sure to properly configure django logging: ```python @@ -240,7 +286,28 @@ django-todo uses pytest exclusively for testing. The best way to run the suite i The previous `tox` system was removed with the v2 release, since we no longer aim to support older Python or Django versions. -# Version History + +## Upgrade Notes + +django-todo 2.0 was rebuilt almost from the ground up, and included some radical changes, including model name changes. As a result, it is *not compatible* with data from django-todo 1.x. If you would like to upgrade an existing installation, try this: + +* Use `./manage.py dumpdata todo --indent 4 > todo.json` to export your old todo data +* Edit the dump file, replacing the old model names `Item` and `List` with the new model names (`Task` and `TaskList`) +* Delete your existing todo data +* Uninstall the old todo app and reinstall +* Migrate, then use `./manage.py loaddata todo.json` to import the edited data + +### Why not provide migrations? + +That was the plan, but unfortunately, `makemigrations` created new tables and dropped the old ones, making this a destructive update. Renaming models is unfortunately not something `makemigrations` can do, and I really didn't want to keep the badly named original models. Sorry! + +### Datepicker + +django-todo no longer references a jQuery datepicker, but defaults to native html5 browser datepicker (not supported by Safari, unforunately). Feel free to implement one of your choosing. + +## Version History + +**2.4.0** Added ability to batch-import tasks via CSV **2.3.0** Implement mail tracking system diff --git a/docs/index.md b/docs/index.md index cc94810..f60533e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -14,6 +14,7 @@ assignment application for Django, designed to be dropped into an existing site * Public-facing submission form for tickets * Mobile-friendly (work in progress) * Separate view for My Tasks (across lists) +* Batch-import tasks via CSV ## Requirements @@ -44,7 +45,7 @@ All tasks are "created by" the current user and can optionally be "assigned to" django-todo v2 makes use of features only available in Django 2.0. It will not work in previous versions. v2 is only tested against Python 3.x -- no guarantees if running it against older versions. -# Installation +## Installation django-todo is a Django app, not a project site. It needs a site to live in. You can either install it into an existing Django project site, or clone the django-todo [demo site (GTD)](https://github.com/shacker/gtd). @@ -132,7 +133,6 @@ The current django-todo version number is available from the [todo package](http python -c "import todo; print(todo.__version__)" - ## Upgrade Notes django-todo 2.0 was rebuilt almost from the ground up, and included some radical changes, including model name changes. As a result, it is *not compatible* with data from django-todo 1.x. If you would like to upgrade an existing installation, try this: @@ -153,7 +153,7 @@ django-todo no longer references a jQuery datepicker, but defaults to native htm ### URLs -Some views and URLs were renamed for logical consistency. If this affects you, see source code and the demo GTD site for reference to the new URL names. +Some views and URLs were renamed in 2.0 for logical consistency. If this affects you, see source code and the demo GTD site for reference to the new URL names. ## Running Tests @@ -166,7 +166,49 @@ django-todo uses pytest exclusively for testing. The best way to run the suite i The previous `tox` system was removed with the v2 release, since we no longer aim to support older Python or Django versions. -# Version History +## Importing Tasks via CSV + +django-todo has the ability to batch-import ("upsert") tasks from a specifically formatted CSV spreadsheet. This ability is provided through both a management command and a web interface. + +**Management Command** + +`./manage.py import_csv -f /path/to/file.csv` + +**Web Importer** + +Link from your navigation to `{url "todo:import_csv"}` + +### Import Rules + +Because data entered via CSV is not going through the same view permissions enforced in the rest of django-todo, and to simplify data dependency logic, and to pre-empt disagreements between django-todo users, the importer will *not* create new users, groups, or task lists. All users, groups, and task lists referenced in your CSV must already exist, and group memberships must be correct. + +Any validation error (e.g. unparse-able dates, incorrect group memberships) **will result in that row being skipped.** + +A report of rows upserted and rows skipped (with line numbers and reasons) is provided at the end of the run. + +### CSV Formatting + +Copy `todo/data/import_example.csv` to another location on your system and edit in a spreadsheet or directly. + +**Do not edit the header row!** + +The first four columns: `'Title', 'Group', 'Task List', 'Created By'` are required -- all others are optional and should work pretty much exactly like manual task entry via the web UI. + +Note: Internally, Tasks are keyed to TaskLists, not to Groups (TaskLists are in Gruops). However, we request the Group in the CSV +because it's possible to have multiple TaskLists with the same name in different groups; i.e. we need it for namespacing and permissions. + +### Upsert Logic + +For each valid row, we need to decide whether to create a new task or update an existing one. django-todo matches on the unique combination of the four required columns. If we find a task that matches those, we *update* the rest of the columns. In other words, if you import a CSV once, then edit the Assigned To for a task and import it again, the original task will be updated with a new assignee (and same for the other columns). + +Otherwise we create a new task. + + +## Version History + +**2.3.0** Added ability to batch-import tasks via CSV + +**2.2.1** Convert task delete and toggle_done views to POST only **2.2.0** Re-instate enforcement of TODO_STAFF_ONLY setting @@ -225,3 +267,5 @@ ALL groups, not just the groups they "belong" to) **0.9.1** - Removed context_processors.py - leftover turdlet **0.9** - First release + + diff --git a/test_settings.py b/test_settings.py index 14068d3..81007c8 100644 --- a/test_settings.py +++ b/test_settings.py @@ -80,7 +80,7 @@ LOGGING = { }, 'django': { 'handlers': ['console'], - 'level': 'DEBUG', + 'level': 'WARNING', 'propagate': True, }, 'django.request': { diff --git a/todo/data/import_example.csv b/todo/data/import_example.csv new file mode 100644 index 0000000..1a68723 --- /dev/null +++ b/todo/data/import_example.csv @@ -0,0 +1,4 @@ +Title,Group,Task List,Created By,Created Date,Due Date,Completed,Assigned To,Note,Priority +Make dinner,Scuba Divers,Web project,shacker,,2019-06-14,No,,Please check with mgmt first,3 +Bake bread,Scuba Divers,Example List,mr_random,2012-03-14,,Yes,,, +Bring dessert,Scuba Divers,Web project,user1,2015-06-248,,,user1,Every generation throws a hero up the pop charts,77 \ No newline at end of file diff --git a/todo/management/commands/import_csv.py b/todo/management/commands/import_csv.py new file mode 100644 index 0000000..da9398f --- /dev/null +++ b/todo/management/commands/import_csv.py @@ -0,0 +1,57 @@ +import sys +from typing import Any +from pathlib import Path + +from django.core.management.base import BaseCommand, CommandParser + +from todo.operations.csv_importer import CSVImporter + + +class Command(BaseCommand): + help = """Import specifically formatted CSV file containing incoming tasks to be loaded. + For specfic format of inbound CSV, see data/import_example.csv. + For documentation on upsert logic and required fields, see README.md. + """ + + def add_arguments(self, parser: CommandParser) -> None: + + parser.add_argument( + "-f", "--file", dest="file", default=None, help="File to to inbound CSV file." + ) + + def handle(self, *args: Any, **options: Any) -> None: + # Need a file to proceed + if not options.get("file"): + print("Sorry, we need a filename to work from.") + sys.exit(1) + + filepath = Path(options["file"]) + + if not filepath.exists(): + print(f"Sorry, couldn't find file: {filepath}") + sys.exit(1) + + # Encoding "utf-8-sig" means "ignore byte order mark (BOM), which Excel inserts when saving CSVs." + with filepath.open(mode="r", encoding="utf-8-sig") as fileobj: + importer = CSVImporter() + results = importer.upsert(fileobj, as_string_obj=True) + + # Report successes, failures and summaries + print() + if results["upserts"]: + for upsert_msg in results["upserts"]: + print(upsert_msg) + + # Stored errors has the form: + # self.errors = [{3: ["Incorrect foo", "Non-existent bar"]}, {7: [...]}] + if results["errors"]: + for error_dict in results["errors"]: + for k, error_list in error_dict.items(): + print(f"\nSkipped CSV row {k}:") + for msg in error_list: + print(f"- {msg}") + + print() + if results["summaries"]: + for summary_msg in results["summaries"]: + print(summary_msg) diff --git a/todo/migrations/0009_priority_optional.py b/todo/migrations/0009_priority_optional.py new file mode 100644 index 0000000..108ec30 --- /dev/null +++ b/todo/migrations/0009_priority_optional.py @@ -0,0 +1,22 @@ +# Generated by Django 2.1.7 on 2019-03-18 23:14 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('todo', '0008_mail_tracker'), + ] + + operations = [ + migrations.AlterModelOptions( + name='task', + options={'ordering': ['priority', 'created_date']}, + ), + migrations.AlterField( + model_name='task', + name='priority', + field=models.PositiveIntegerField(blank=True, null=True), + ), + ] diff --git a/todo/models.py b/todo/models.py index 67e042f..01bb19a 100644 --- a/todo/models.py +++ b/todo/models.py @@ -82,7 +82,7 @@ class Task(models.Model): on_delete=models.CASCADE, ) note = models.TextField(blank=True, null=True) - priority = models.PositiveIntegerField() + priority = models.PositiveIntegerField(blank=True, null=True) # Has due date for an instance of this object passed? def overdue_status(self): @@ -115,7 +115,7 @@ class Task(models.Model): self.delete() class Meta: - ordering = ["priority"] + ordering = ["priority", "created_date"] class Comment(models.Model): diff --git a/todo/operations/__init__.py b/todo/operations/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/todo/operations/csv_importer.py b/todo/operations/csv_importer.py new file mode 100644 index 0000000..9fb3b18 --- /dev/null +++ b/todo/operations/csv_importer.py @@ -0,0 +1,197 @@ +import codecs +import csv +import datetime +import logging + +from django.contrib.auth import get_user_model +from django.contrib.auth.models import Group + +from todo.models import Task, TaskList + +log = logging.getLogger(__name__) + + +class CSVImporter: + """Core upsert functionality for CSV import, for re-use by `import_csv` management command, web UI and tests. + Supplies a detailed log of what was and was not imported at the end. See README for usage notes. + """ + + def __init__(self): + self.errors = [] + self.upserts = [] + self.summaries = [] + self.line_count = 0 + self.upsert_count = 0 + + def upsert(self, fileobj, as_string_obj=False): + """Expects a file *object*, not a file path. This is important because this has to work for both + the management command and the web uploader; the web uploader will pass in in-memory file + with no path! + + Header row is: + Title, Group, Task List, Created Date, Due Date, Completed, Created By, Assigned To, Note, Priority + """ + + if as_string_obj: + # fileobj comes from mgmt command + csv_reader = csv.DictReader(fileobj) + else: + # fileobj comes from browser upload (in-memory) + csv_reader = csv.DictReader(codecs.iterdecode(fileobj, "utf-8")) + + # DI check: Do we have expected header row? + header = csv_reader.fieldnames + expected = [ + "Title", + "Group", + "Task List", + "Created By", + "Created Date", + "Due Date", + "Completed", + "Assigned To", + "Note", + "Priority", + ] + if header != expected: + self.results.get("summaries").append( + f"Inbound data does not have expected columns.\nShould be: {expected}" + ) + return self.results + + for row in csv_reader: + self.line_count += 1 + + newrow = self.validate_row(row) + if newrow: + # newrow at this point is fully validated, and all FK relations exist, + # e.g. `newrow.get("Assigned To")`, is a Django User instance. + assignee = newrow.get("Assigned To") if newrow.get("Assigned To") else None + created_date = newrow.get("Created Date") if newrow.get("Created Date") else datetime.datetime.today() + due_date = newrow.get("Due Date") if newrow.get("Due Date") else None + priority = newrow.get("Priority") if newrow.get("Priority") else None + + obj, created = Task.objects.update_or_create( + created_by=newrow.get("Created By"), + task_list=newrow.get("Task List"), + title=newrow.get("Title"), + defaults={ + "assigned_to": assignee, + "completed": newrow.get("Completed"), + "created_date": created_date, + "due_date": due_date, + "note": newrow.get("Note"), + "priority": priority, + }, + ) + self.upsert_count += 1 + msg = ( + f'Upserted task {obj.id}: "{obj.title}"' + f' in list "{obj.task_list}" (group "{obj.task_list.group}")' + ) + self.upserts.append(msg) + + self.summaries.append(f"Processed {self.line_count} CSV rows") + self.summaries.append(f"Upserted {self.upsert_count} rows") + self.summaries.append(f"Skipped {self.line_count - self.upsert_count} rows") + + return {"summaries": self.summaries, "upserts": self.upserts, "errors": self.errors} + + def validate_row(self, row): + """Perform data integrity checks and set default values. Returns a valid object for insertion, or False. + Errors are stored for later display. Intentionally not broken up into separate validator functions because + there are interdpendencies, such as checking for existing `creator` in one place and then using + that creator for group membership check in others.""" + + row_errors = [] + + # ####################### + # Task creator must exist + if not row.get("Created By"): + msg = f"Missing required task creator." + row_errors.append(msg) + + creator = get_user_model().objects.filter(username=row.get("Created By")).first() + if not creator: + msg = f"Invalid task creator {row.get('Created By')}" + row_errors.append(msg) + + # ####################### + # If specified, Assignee must exist + assignee = None # Perfectly valid + if row.get("Assigned To"): + assigned = get_user_model().objects.filter(username=row.get("Assigned To")) + if assigned.exists(): + assignee = assigned.first() + else: + msg = f"Missing or invalid task assignee {row.get('Assigned To')}" + row_errors.append(msg) + + # ####################### + # Group must exist + try: + target_group = Group.objects.get(name=row.get("Group")) + except Group.DoesNotExist: + msg = f"Could not find group {row.get('Group')}." + row_errors.append(msg) + target_group = None + + # ####################### + # Task creator must be in the target group + if creator and target_group not in creator.groups.all(): + msg = f"{creator} is not in group {target_group}" + row_errors.append(msg) + + # ####################### + # Assignee must be in the target group + if assignee and target_group not in assignee.groups.all(): + msg = f"{assignee} is not in group {target_group}" + row_errors.append(msg) + + # ####################### + # Task list must exist in the target group + try: + tasklist = TaskList.objects.get(name=row.get("Task List"), group=target_group) + row["Task List"] = tasklist + except TaskList.DoesNotExist: + msg = f"Task list {row.get('Task List')} in group {target_group} does not exist" + row_errors.append(msg) + + # ####################### + # Validate Dates + datefields = ["Due Date", "Created Date"] + for datefield in datefields: + datestring = row.get(datefield) + if datestring: + valid_date = self.validate_date(datestring) + if valid_date: + row[datefield] = valid_date + else: + msg = f"Could not convert {datefield} {datestring} to valid date instance" + row_errors.append(msg) + + # ####################### + # Group membership checks have passed + row["Created By"] = creator + row["Group"] = target_group + if assignee: + row["Assigned To"] = assignee + + # Set Completed + row["Completed"] = (row["Completed"] == "Yes") + + # ####################### + if row_errors: + self.errors.append({self.line_count: row_errors}) + return False + + # No errors: + return row + + def validate_date(self, datestring): + """Inbound date string from CSV translates to a valid python date.""" + try: + date_obj = datetime.datetime.strptime(datestring, "%Y-%m-%d") + return date_obj + except ValueError: + return False diff --git a/todo/templates/todo/import_csv.html b/todo/templates/todo/import_csv.html new file mode 100644 index 0000000..121925b --- /dev/null +++ b/todo/templates/todo/import_csv.html @@ -0,0 +1,85 @@ +{% extends "todo/base.html" %} +{% load static %} + +{% block title %}Import CSV{% endblock %} + +{% block content %} +
+ Batch-import tasks by uploading a specifically-formatted CSV. + See documentation for formatting rules. + Successs and failures will be reported here. +
+ + {% if results %} ++ Summary: +
++ Upserts (tasks created or updated): +
++ Errors (tasks NOT created or updated): +
+