Exception to cookie consent check for view/static

Should mitigate internal server errors if SMTP server fails.

README.md

@@ -29,12 +29,11 @@ To set up the server, consult some of the comprehensive guides on various hostin
 Here is a [good starting point on setting up a server](https://www.digitalocean.com/community/tutorials/initial-server-setup-with-ubuntu-22-04).
 
 To install Docker and Docker Compose, consult the respective documentation:
+
 - [Install on Ubuntu](https://docs.docker.com/engine/install/ubuntu/) or [Install on Debian](https://docs.docker.com/engine/install/debian/)
 - Docker Compose should be installed as part of that.
 
-```
+> At the time of writing, there has been an upgrade to Docker and Docker Compose, meaning the syntax below might be different between versions.
-At the time of writing, there has been an upgrade to Docker and Docker Compose, meaning the syntax below might be different between versions.
-```
 
 Check if Git is installed on your server using the `git --version` command.
 If it isn't installed, install it.
@@ -72,13 +71,13 @@ Also make sure that the various entries for usernames and passwords match.
 
 There are some values in the following four files you will need to configure to reflect the domain you are installing this app.
 
-```
+```sh
 # .env
 
 SERVER_NAME= # URL where this will be hosted.
 ```
 
-```
+```sh
 # install-script.sh
 
 domains=(example.org www.example.org)
@@ -87,7 +86,7 @@ email="" # Adding a valid address is strongly recommended
 
 Substitute the domain name `domain_name` in the two file paths in the following file:
 
-```
+```sh
 # nginx/ssl.conf
 
 ssl_certificate /etc/letsencrypt/live/domain_name/fullchain.pem;
@@ -95,9 +94,9 @@ ssl_certificate_key /etc/letsencrypt/live/domain_name/privkey.pem;
 ...
 ```
 
-And **six** locations in the following file, two for the regular version of the domain and two for the www version:
+And **six** locations in the following file, two for the regular version of the domain and four for the www version (remember to keep the www. prefix where present):
 
-```
+```nginx
 # nginx/conf.d/ref-test-app.conf
 
 server {
@@ -140,9 +139,9 @@ This will be set up automatically.
 However, there is a specific chicken-and-egg problem as the web server, Nginx, won't run without certificates, Certbot, the certificate generator, won't run without the web server.
 So to solve this, there is an automation script we can run that will set up a dummy certificate and then issue the appropriate certificates for us.
 
-```
+```sh
-$ chmod +x install-script.sh
+chmod +x install-script.sh
-$ sudo ./install-script.sh
+sudo ./install-script.sh
 ```
 
 This will take a long time to run the first time because it will try and generate a fairly sizeable cypher.

REFERENCES.md

@@ -6,11 +6,6 @@
 
 - [Docker Compose Reference](https://docs.docker.com/compose/compose-file/compose-file-v3/)
 
-### MongoDB/PyMongo
-
-- [MongoDB Shell Commands](https://docs.mongodb.com/manual/reference/)
-- [PyMongo Driver](https://pymongo.readthedocs.io/en/stable/)
-
 ## Source Code
 
 - [MongoDB Docker Image entrypoint shell script](https://github.com/docker-library/mongo/blob/master/5.0/docker-entrypoint.sh) (Context: Tried to replicate the command to create a new user in the original entrypoint script in the custom initialisation script in this app.)
@@ -23,15 +18,6 @@
 - [Tables](https://www.blog.pythonlibrary.org/2017/12/14/flask-101-adding-editing-and-displaying-data/)
 - [Tables, but interactive](https://blog.miguelgrinberg.com/post/beautiful-interactive-tables-for-your-flask-templates)
 
-## Stack Exchange/Overflow
-
-### MongoDB
-
-- [Creating MongoDB Database on Container Start](https://stackoverflow.com/questions/42912755/how-to-create-a-db-for-mongodb-container-on-start-up)
-- [Passing Environment Variables to Docker Container Entrypoint](https://stackoverflow.com/questions/64606674/how-can-i-pass-environment-variables-to-mongo-docker-entrypoint-initdb-d)
-- [Integrating Flask-Login with MongoDB](https://stackoverflow.com/questions/54992412/flask-login-usermixin-class-with-a-mongodb) (**This does not work with the app as is, and is possibly something that needs more research and development in the future**)
-- [Setting up a Postfix email notification system](https://medium.com/@vietgoeswest/a-simple-outbound-email-service-for-your-app-in-15-minutes-cc4da70a2af7)
-
 ## YouTube Tutorials
 
 ### General Flask Introduction
@@ -72,7 +58,7 @@ A much simpler and more rudimentary introduction to Flask and MongoDB.
 - [Build a User Login System with `flask-login`, `flask-wtforms`, `flask-bootstrap`, and `flask-sqlalchemy`](https://www.youtube.com/watch?v=8aTnmsDMldY)
 
 A much more robust method that uses the various Flask modules to make a more powerful framework.
-Uses SQL rather than MongoDB.
+Uses SQL.
 
 ### Flask techniques
 
@@ -80,4 +66,4 @@ Uses SQL rather than MongoDB.
 
 ### Flask handling file uploads
 
-- [Handlin File Uploads](https://blog.miguelgrinberg.com/post/handling-file-uploads-with-flask)
+- [Handling File Uploads](https://blog.miguelgrinberg.com/post/handling-file-uploads-with-flask)

docker-compose.yml

@@ -10,11 +10,12 @@ services:
     volumes:
       - ./certbot:/etc/letsencrypt:ro
       - ./nginx:/etc/nginx
-      - ./src/html:/usr/share/nginx/html/
+      - ./src/html/robots.txt:/usr/share/nginx/html/robots.txt:ro
-      - ./ref-test/app/editor/static:/usr/share/nginx/html/admin/editor/static
+      - ./ref-test/app/root:/usr/share/nginx/html/root:ro
-      - ./ref-test/app/admin/static:/usr/share/nginx/html/admin/static
+      - ./ref-test/app/admin/static:/usr/share/nginx/html/admin/static:ro
-      - ./ref-test/app/quiz/static:/usr/share/nginx/html/quiz/static
+      - ./ref-test/app/editor/static:/usr/share/nginx/html/editor/static:ro
-      - ./ref-test/app/root:/usr/share/nginx/html/root
+      - ./ref-test/app/quiz/static:/usr/share/nginx/html/quiz/static:ro
+      - ./ref-test/app/view/static:/usr/share/nginx/html/view/static:ro
     ports:
       - 80:80
       - 443:443

nginx/conf.d/ref-test-app.conf

@@ -20,6 +20,11 @@ server {
     include	/etc/nginx/certbot-challenge.conf;
 
     # Define locations for static files to be served by Nginx
+    location ^~ /root/ {
+        include  /etc/nginx/mime.types;
+        alias /usr/share/nginx/html/root/;
+    }
+
     location ^~ /quiz/static/  {
         include  /etc/nginx/mime.types;
         alias /usr/share/nginx/html/quiz/static/;
@@ -32,12 +37,12 @@ server {
 
     location ^~ /admin/editor/static/  {
         include  /etc/nginx/mime.types;
-        alias /usr/share/nginx/html/admin/editor/static/;
+        alias /usr/share/nginx/html/editor/static/;
     }
 
     location ^~ /admin/view/static/  {
         include  /etc/nginx/mime.types;
-        alias /usr/share/nginx/html/admin/view/static/;
+        alias /usr/share/nginx/html/view/static/;
     }
 
     # Proxy to the main app for all other requests

ref-test/app/__init__.py

@@ -30,7 +30,7 @@ def create_app():
     def _check_cookie_consent():
         if request.cookies.get('cookie_consent'):
             return
-        if any([ request.path.startswith(x) for x in [ '/admin/static/', '/root/', '/quiz/static', '/cookies/', '/admin/editor/static' ] ]):
+        if any([ request.path.startswith(x) for x in [ '/admin/static/', '/root/', '/quiz/static', '/cookies/', '/admin/editor/static', '/admin/view/static' ] ]):
             return
         flash(f'<strong>Cookie Consent</strong>: This web site only stores minimal, functional cookies. It does not store any tracking information. By using this site, you consent to this use of cookies. For more information, see our <a href="{url_for("views._privacy")}">privacy policy</a>.', 'cookie_alert')
 

ref-test/app/models/dataset.py

@@ -5,6 +5,7 @@ from ..tools.logs import write
 from flask import flash
 from flask import current_app as app
 from flask_login import current_user
+from sqlalchemy.exc import SQLAlchemyError
 from werkzeug.utils import secure_filename
 
 from datetime import datetime
@@ -45,7 +46,12 @@ class Dataset(db.Model):
         for dataset in Dataset.query.all():
             dataset.default = False
         self.default = True
+        try:
             db.session.commit()
+        except SQLAlchemyError as exception:
+            db.session.rollback()
+            write('system.log', f'Database error when setting default dataset {self.id}: {exception}')
+            return False, f'Database error {exception}.'
         write('system.log', f'Dataset {self.id} set as default by {current_user.get_username()}.')
         flash(message='Dataset set as default.', category='success')
         return True, f'Dataset set as default.'
@@ -63,9 +69,14 @@ class Dataset(db.Model):
         filename = secure_filename('.'.join([self.id,'json']))
         data = Path(app.config.get('DATA'))
         file_path = path.join(data, 'questions', filename)
-        remove(file_path)
+        try:
             db.session.delete(self)
             db.session.commit()
+        except SQLAlchemyError as exception:
+            db.session.rollback()
+            write('system.log', f'Database error when trying to delete dataset {self.id}: {exception}')
+            return False, f'Database error: {exception}'
+        remove(file_path)
         return True, 'Dataset deleted.'
     
     def create(self, data:list, default:bool=False):
@@ -78,8 +89,13 @@ class Dataset(db.Model):
         self.creator = current_user
         if default: self.make_default()
         write('system.log', f'New dataset {self.get_name()} added by {current_user.get_username()}.')
+        try:
             db.session.add(self)
             db.session.commit()
+        except SQLAlchemyError as exception:
+            db.session.rollback()
+            write('system.log', f'Database error when trying to crreate dataset {self.id}: {exception}')
+            return False, f'Database error: {exception}'
         return True, 'Dataset created.'
 
     def check_file(self):
@@ -103,6 +119,11 @@ class Dataset(db.Model):
             dump(data, file, indent=2)
         write('system.log', f'Dataset {self.id} edited by {current_user.get_username()}.')
         flash(f'Dataset {self.get_name()} successfully edited.', 'success')
+        try:
             db.session.add(self)
             db.session.commit()
+        except SQLAlchemyError as exception:
+            db.session.rollback()
+            write('system.log', f'Database error when trying to update dataset {self.id}: {exception}')
+            return False, f'Database error: {exception}'
         return True, 'Dataset successfully edited.'

ref-test/app/models/entry.py

@@ -6,6 +6,8 @@ from .test import Test
 
 from flask_login import current_user
 from flask_mail import Message
+from smtplib import SMTPException
+from sqlalchemy.exc import SQLAlchemyError
 
 from datetime import datetime, timedelta
 from uuid import uuid4
@@ -69,23 +71,32 @@ class Entry(db.Model):
 
     def ready(self):
         self.generate_id()
+        try:
             db.session.add(self)
             db.session.commit()
-        write('tests.log', f'New test ready for {self.get_first_name()} {self.get_surname()}.')
+        except SQLAlchemyError as exception:
+            db.session.rollback()
+            write('system.log', f'Database error when preparing new entry for {self.get_surname()}, {self.get_first_name()}: {exception}')
+            return False, f'Database error: {exception}'
+        write('tests.log', f'New test ready for {self.get_surname()}, {self.get_first_name()} with id {self.id}.')
         return True, f'Test ready.'
 
     def start(self):
         self.start_time = datetime.now()
         self.status = 'started'
-        write('tests.log', f'Test started by {self.get_first_name()} {self.get_surname()}.')
+        try:
             db.session.commit()
+        except SQLAlchemyError as exception:
+            db.session.rollback()
+            write('system.log', f'Database error when starting test for {self.get_surname()}, {self.get_first_name()}: {exception}')
+            return False, f'Database error: {exception}'
+        write('tests.log', f'Test started by {self.get_surname()}, {self.get_first_name()} with id {self.id}.')
         return True, f'New test started with id {self.id}.'
 
     def complete(self, answers:dict=None, result:dict=None):
         self.end_time = datetime.now()
         self.answers = answers
         self.result = result
-        write('tests.log', f'Test completed by {self.get_first_name()} {self.get_surname()}.')
         delta = timedelta(minutes=int(0 if self.test.time_limit is None else self.test.time_limit)+1)
         if not self.test.time_limit or self.end_time <= self.start_time + delta:
             self.status = 'completed'
@@ -93,7 +104,13 @@ class Entry(db.Model):
         else:
             self.status = 'late'
             self.valid = False
+        try:
             db.session.commit()
+        except SQLAlchemyError as exception:
+            db.session.rollback()
+            write('system.log', f'Database error when submitting entry for {self.get_surname()}, {self.get_first_name()}: {exception}')
+            return False, f'Database error: {exception}'
+        write('tests.log', f'Test completed by {self.get_surname()}, {self.get_first_name()} with id {self.id}.')
         return True, f'Test entry completed for id {self.id}.'
 
     def validate(self):
@@ -101,15 +118,25 @@ class Entry(db.Model):
         if self.status == 'started': return False, 'The entry is still pending.'
         self.valid = True
         self.status = 'completed'
+        try:
             db.session.commit()
+        except SQLAlchemyError as exception:
+            db.session.rollback()
+            write('system.log', f'Database error when validating entry {self.id}: {exception}')
+            return False, f'Database error: {exception}'
         write('system.log', f'The entry {self.id} has been validated by {current_user.get_username()}.')
         return True, f'The entry {self.id} has been validated.'
 
     def delete(self):
         id = self.id
         name = f'{self.get_first_name()} {self.get_surname()}'
+        try:
             db.session.delete(self)
             db.session.commit()
+        except SQLAlchemyError as exception:
+            db.session.rollback()
+            write('system.log', f'Database error when deleting entry {id}: {exception}')
+            return False, f'Database error: {exception}'
         write('system.log', f'The entry {id} by {name} has been deleted by {current_user.get_username()}.')
         return True, 'Entry deleted.'
     
@@ -174,4 +201,7 @@ class Entry(db.Model):
             <p>Best wishes, <br/> SKA Refereeing</p>
             """
         )
+        try: 
             mail.send(email)
+        except SMTPException as exception:
+            write('system.log', f'SMTP Error when trying to notify results to {self.get_surname()}, {self.get_first_name()} with error: {exception}')

ref-test/app/models/test.py

@@ -3,6 +3,7 @@ from ..tools.forms import JsonEncodedDict
 from ..tools.logs import write
 
 from flask_login import current_user
+from sqlalchemy.exc import SQLAlchemyError
 
 from datetime import date, datetime
 import secrets
@@ -52,16 +53,25 @@ class Test(db.Model):
             errors.append('The expiry date cannot be before the start date.')
         if errors:
             return False, errors
+        try:
             db.session.add(self)
             db.session.commit()
+        except SQLAlchemyError as exception:
+            db.session.rollback()
+            write('system.log', f'Database error when creating test {self.get_code()}: {exception}')
+            return False, f'Database error: {exception}'
         write('system.log', f'Test with code {self.get_code()} created by {current_user.get_username()}.')
         return True, f'Test with code {self.get_code()} has been created.'
     
     def delete(self):
-        code = self.code
         if self.entries: return False, f'Cannot delete a test with submitted entries.'
         db.session.delete(self)
+        try:
             db.session.commit()
+        except SQLAlchemyError as exception:
+            db.session.rollback()
+            write('system.log', f'Database error when deleting test {self.get_code()}: {exception}')
+            return False, f'Database error: {exception}'
         write('system.log', f'Test with code {self.get_code()} has been deleted by {current_user.get_username()}.')
         return True, f'Test with code {self.get_code()} has been deleted.'
     
@@ -69,7 +79,12 @@ class Test(db.Model):
         now = datetime.now()
         if self.start_date.date() > now.date():
             self.start_date = now
+            try:
                 db.session.commit()
+            except SQLAlchemyError as exception:
+                db.session.rollback()
+                write('system.log', f'Database error when launching test {self.get_code()}: {exception}')
+                return False, f'Database error: {exception}'
             write('system.log', f'Test with code {self.get_code()} has been started by {current_user.get_username()}.')
             return True,  f'Test with code {self.get_code()} has been started.'
         return False, f'Test with code {self.get_code()} has already started.'
@@ -78,7 +93,12 @@ class Test(db.Model):
         now = datetime.now()
         if self.end_date >= now:
             self.end_date = now
+            try:
                 db.session.commit()
+            except SQLAlchemyError as exception:
+                db.session.rollback()
+                write('system.log', f'Database error when closing test {self.get_code()}: {exception}')
+                return False, f'Database error: {exception}'
             write('system.log', f'Test with code {self.get_code()} ended by {current_user.get_username()}.')
             return True, f'Test with code {self.get_code()} has been ended.'
         return False, f'Test with code {self.get_code()} has already ended.'
@@ -88,7 +108,12 @@ class Test(db.Model):
         code = secrets.token_hex(3).lower()
         adjustments[code] = time
         self.adjustments = adjustments
+        try:
             db.session.commit()
+        except SQLAlchemyError as exception:
+            db.session.rollback()
+            write('system.log', f'Database error when adding adjustment to test {self.get_code()}: {exception}')
+            return False, f'Database error: {exception}'
         write('system.log', f'Time adjustment for {time} minutes with code {code} added to test {self.get_code()} by {current_user.get_username()}.')
         return True, f'Time adjustment for {time} minutes added to test {self.get_code()}. This can be accessed using the user code {code.upper()}.'
 
@@ -96,7 +121,12 @@ class Test(db.Model):
         if not self.adjustments: return False, f'There are no adjustments configured for test {self.get_code()}.'
         self.adjustments.pop(code)
         if not self.adjustments: self.adjustments = None
+        try:
             db.session.commit()
+        except SQLAlchemyError as exception:
+            db.session.rollback()
+            write('system.log', f'Database error when deleting adjustment from test {self.get_code()}: {exception}')
+            return False, f'Database error: {exception}'
         write('system.log', f'Time adjustment for with code {code} has been removed from test {self.get_code()} by {current_user.get_username()}.')
         return True, f'Time adjustment for with code {code} has been removed from test {self.get_code()}.'
 
@@ -105,6 +135,11 @@ class Test(db.Model):
         if start_date: self.start_date = start_date
         if end_date: self.end_date = end_date
         if time_limit is not None: self.time_limit = time_limit
+        try:
             db.session.commit()
+        except SQLAlchemyError as exception:
+            db.session.rollback()
+            write('system.log', f'Database error when updating test {self.get_code()}: {exception}')
+            return False, f'Database error: {exception}'
         write('system.log', f'Test with code {self.get_code()} has been updated by user {current_user.get_username()}.')
         return True, f'Test with code {self.get_code()} has been updated by.'

ref-test/app/models/user.py

@@ -6,6 +6,8 @@ from flask import flash, jsonify, session
 from flask.helpers import url_for
 from flask_login import current_user, login_user, logout_user, UserMixin
 from flask_mail import Message
+from smtplib import SMTPException
+from sqlalchemy.exc import SQLAlchemyError
 from werkzeug.security import check_password_hash, generate_password_hash
 
 import secrets
@@ -60,8 +62,13 @@ class User(UserMixin, db.Model):
             if user.get_username() == self.get_username(): return False, f'Username {self.get_username()} already in use.'
             if user.get_email() == self.get_email(): return False, f'Email address {self.get_email()} already in use.'
         self.set_password(password=password)
+        try:
             db.session.add(self)
             db.session.commit()
+        except SQLAlchemyError as exception:
+            db.session.rollback()
+            write('system.log', f'Database error when registering user {self.get_username()}: {exception}')
+            return False, f'Database error: {exception}'
         write('users.log', f'User \'{self.get_username()}\' was created with id \'{self.id}\'.')
         if notify:
             email = Message(
@@ -90,7 +97,10 @@ class User(UserMixin, db.Model):
                 <p>SKA Refereeing</p>
                 """
             )
+            try:
                 mail.send(email)
+            except SMTPException as exception:
+                write('system.log', f'SMTP Error while trying to notify new user account creation to {self.get_username()} with error: {exception}')
         return True, f'User {self.get_username()} was created successfully.'
 
     def login(self, remember:bool=False):
@@ -109,7 +119,6 @@ class User(UserMixin, db.Model):
         self.set_password(new_password)
         self.reset_token = secrets.token_urlsafe(16)
         self.verification_token = secrets.token_urlsafe(16)
-        db.session.commit()
         email = Message(
             subject='RefTest | Password Reset',
             recipients=[self.get_email()],
@@ -142,22 +151,39 @@ class User(UserMixin, db.Model):
             <p>SKA Refereeing</p>
             """
         )
+        try:
             mail.send(email)
-        print('Password', new_password)
+        except SMTPException as exception:
-        print('Reset Token', self.reset_token)
+            write('system.log', f'SMTP Error while trying to reset password for {self.get_username()} with error: {exception}')
-        print('Verification Token', self.verification_token)
+            db.session.rollback()
-        print('Reset Link', f'{url_for("admin._reset", token=self.reset_token, verification=self.verification_token, _external=True)}')
+            return jsonify({'error': f'SMTP Error: {exception}'}), 500
+        try:
+            db.session.commit()
+        except SQLAlchemyError as exception:
+            db.session.rollback()
+            write('system.log', f'Database error when resetting password for user {self.get_username()}: {exception}')
+            return False, f'Database error: {exception}'
         return jsonify({'success': 'Your password reset link has been generated.'}), 200
 
     def clear_reset_tokens(self):
         self.reset_token = self.verification_token = None
+        try:
             db.session.commit()
+        except SQLAlchemyError as exception:
+            db.session.rollback()
+            write('system.log', f'Database error when resetting clearing reset tokens for user {self.get_username()}: {exception}')
+            return False, f'Database error: {exception}'
     
     def delete(self, notify:bool=False):
         username = self.get_username()
         email_address = self.get_email()
+        try:
             db.session.delete(self)
             db.session.commit()
+        except SQLAlchemyError as exception:
+            db.session.rollback()
+            write('system.log', f'Database error when deleting user {self.get_username()}: {exception}')
+            return False, f'Database error: {exception}'
         message = f'User \'{username}\' was deleted by \'{current_user.get_username()}\'.'
         write('users.log', message)
         if notify:
@@ -182,7 +208,10 @@ class User(UserMixin, db.Model):
                 <p>SKA Refereeing</p>
                 """
             )
+            try:
                 mail.send(email)
+            except SMTPException as exception:
+                write('system.log', f'SMTP Error when trying to delete account {username} with error: {exception}')
         return True, message
 
     def update(self, password:str=None, email:str=None, notify:bool=False):
@@ -193,7 +222,12 @@ class User(UserMixin, db.Model):
             for entry in User.query.all():
                 if entry.get_email() == email and not entry == self: return False, f'The email address {email} is already in use.'
             self.set_email(email)
+        try:
             db.session.commit()
+        except SQLAlchemyError as exception:
+            db.session.rollback()
+            write('system.log', f'Database error when updating user {self.get_username()}: {exception}')
+            return False, f'Database error: {exception}'
         _current_user = current_user.get_username() if current_user.is_authenticated else 'anonymous'
         write('system.log', f'Information for user {self.get_username()} has been updated by {_current_user}.')
         if notify:
@@ -223,5 +257,8 @@ class User(UserMixin, db.Model):
                 <p>SKA Refereeing</p>
                 """
             )
+            try:
                 mail.send(message)
+            except SMTPException as exception:
+                write('system.log', f'SMTP Error when trying to update account {self.get_username()} with error: {exception}')
         return True, f'Account {self.get_username()} has been updated.'

ref-test/requirements.txt

@@ -1,13 +1,13 @@
-blinker==1.4
+blinker==1.5
-cffi==1.15.0
+cffi==1.15.1
 click==8.1.3
-cryptography==37.0.2
+cryptography==37.0.4
 dnspython==2.2.1
-dominate==2.6.0
+dominate==2.7.0
 email-validator==1.2.1
-Flask==2.1.2
+Flask==2.2.2
 Flask-Bootstrap==3.3.7.1
-Flask-Login==0.6.1
+Flask-Login==0.6.2
 Flask-Mail==0.9.1
 Flask-SQLAlchemy==2.5.1
 Flask-WTF==1.0.1
@@ -20,8 +20,8 @@ MarkupSafe==2.1.1
 pycparser==2.21
 python-dotenv==0.20.0
 six==1.16.0
-SQLAlchemy==1.4.37
+SQLAlchemy==1.4.40
-SQLAlchemy-Utils==0.38.2
+SQLAlchemy-Utils==0.38.3
 visitor==0.1.3
-Werkzeug==2.1.2
+Werkzeug==2.2.2
 WTForms==3.0.1