Merge pull request #3304 from cwivagg/cwivagg/matomo_20220618

Cwivagg/matomo 20220618

roles/0-init/tasks/validate_vars.yml

@@ -63,7 +63,8 @@
 #
 # 2020-11-04: Fix validation of 5 [now 4] core dependencies, for ./runrole etc
 
-- name: Set vars_checklist for 43 + 43 + 39 vars ("XYZ_install" + "XYZ_enabled" + "XYZ_installed") to be checked
+
+- name: Set vars_checklist for 44 + 44 + 40 vars ("XYZ_install" + "XYZ_enabled" + "XYZ_installed") to be checked
   set_fact:
     vars_checklist:
       - hostapd
@@ -100,6 +101,7 @@
       - osm_vector_maps
       - transmission
       - awstats
+      - matomo
       - monit
       - munin
       - phpmyadmin

roles/8-mgmt-tools/tasks/main.yml

@@ -12,7 +12,12 @@
   include_role:
     name: awstats
   when: awstats_install
-  
+
+- name: MATOMO
+  include_role:
+    name: matomo
+  when: matomo_install
+
 - name: MONIT
   include_role:
     name: monit

roles/matomo/README.adoc

@@ -0,0 +1,69 @@
+= Matomo README
+
+https://matomo.org/[Matomo] is a web analytics alternative to Google Analytics, emphasizing privacy and data ownership, that you can use with https://internet-in-a-box.org[Internet-in-a-Box] (IIAB).
+
+== Install it
+
+Prior to installing Matomo with IIAB, the default URL (http://box.lan/matomo) can be customized in https://wiki.iiab.io/go/FAQ#What_is_local_vars.yml_and_how_do_I_customize_it.3F[/etc/iiab/local_vars.yml]
+
+One way to do that is by changing these 2 lines:
+
+----
+iiab_hostname: box
+iiab_domain: lan
+----
+
+Or, you can change the Matomo URL by putting your IIAB IP Address in a line like:
+
+----
+matomo_host_url: http://192.168.0.199
+----
+
+Either way, consider setting a Matomo username and password using lines like:
+
+----
+matomo_db_user: Admin
+matomo_db_pass: changeme
+----
+
+Also ensure that your `/etc/iiab/local_vars.yml` contains these lines:
+
+----
+matomo_install: True
+matomo_enabled: True
+----
+
+_Finally, continue to https://download.iiab.io[install IIAB], e.g. by running `sudo iiab`, until software installation is complete._
+
+== Use it
+
+Log in to your IIAB's full Matomo URL, e.g. http://box.lan/matomo, as arranged above.
+
+WARNING: If your IIAB URL is *not* http://box.lan, you may run into a big orange warning from Matomo that it has been configured to run from a different address. Here are the steps to fix this problem.
+
+1. Copy the IP address listed in the box below "How do I fix this problem and how do I log in again?" For example, I see 'trusted_hosts[] = "192.168.64.10"', so I copy "192.168.64.10".
+2. Run "sudo nano /library/www/matomo/config/config.ini.php" to edit Matomo's config file.
+3. Paste or type the IP address from Step 1 to replace "box.lan" in the trusted_hosts line, which should be about line 13. When I'm done, my line 13 says 'trusted_hosts[] = "192.168.64.10"' instead of 'trusted_hosts[] = "box.lan"'.
+4. Refresh the Matomo homepage and the warning should be gone.
+
+Take a look at Matomo's official guides to further set this up: https://matomo.org/guides/
+
+WARNING: Matomo won't show any traffic statistics until after 1 day or reboot (which are the events that trigger the log scraper!)
+
+=== Getting Started
+
+Matomo is developed with commercial websites in mind. After navigating to http://box.lan/matomo and logging in with the user name and password you set above, you will see a variety of references to revenue, marketplaces, and other terms focused on commercialization and advertising. Don't worry about that.
+
+The heart of Matomo's value for you is in the navigation bar on the left side of the page. Click on "Visitors" and then below "Visitors", "Overview", to see how many different users are visiting your site. The top of the page will show a graph of how many visits occur on each day (although your device can't keep track of time when it is off and has no connection to the Internet, so this graph might not be perfectly accurate). Below the graph, you'll see some overall statistics, like how many unique visitors you've had. Matomo thinks of visitors in terms of devices, so it won't know if two people are connecting to Internet in a Box using the same phone. There are several other interesting statistics here, like the average visit duration, or average time your visitors are spending using Internet in a Box.
+
+Below the "Visitors" button is a second button, "Behavior". Click on the "Pages" button after clicking "Behavior" and you can see the various pages that have been visited by your users. You may not see activity from the most recent day, since Matomo only updates its records once per day.
+
+=== IIAB Tips, Tricks, and Gotchas
+
+1. If your Internet-in-a-Box setup is without power and Internet access, it will not be able to keep time correctly. This is okay! But it means that the time-of-visit information in Matomo will not be correct.
+
+2. One thing Matomo can't track correctly is navigation within Khan Academy pages. If your users are spending a lot of time here, it won't be visible in the Matomo statistics.
+
+== Credits
+
+Carl Wivagg

roles/matomo/defaults/main.yml

@@ -0,0 +1,18 @@
+# matomo_install: True
+# matomo_enabled: True
+
+# All above are set in: github.com/iiab/iiab/blob/master/vars/default_vars.yml
+# If nec, change them by editing /etc/iiab/local_vars.yml prior to installing!
+
+matomo_dl_url: https://builds.matomo.org/matomo.tar.gz
+matomo_path: "{{ doc_base }}"    # e.g. /library/www
+
+matomo_db_name: matomodb
+matomo_db_user: Admin
+matomo_db_pass: changeme
+
+#matomo_host_url: http://{{ ansible_default_ipv4.address }}
+matomo_host_url: http://{{ iiab_hostname }}.{{ iiab_domain }}    # e.g. http://box.lan
+matomo_full_url: "{{ matomo_host_url }}/matomo/"
+
+matomo_cronjob: "sudo python3 {{ matomo_path }}/matomo/misc/log-analytics/import_logs.py --url={{ matomo_full_url }} --idsite=1 --recorders=4 --enable-http-errors --enable-http-redirects --enable-static --enable-bots /var/log/nginx/access.log"

roles/matomo/tasks/install.yml

@@ -0,0 +1,196 @@
+# The sections of code interacting with the Matomo website are modified from code found at https://git.coop/webarch/matomo/. This code is distributed under
+# Version 3 of the GNU General Public License. We modified this code and applied it here in April 2022. The derived sections correspond to the tasks running
+# from "HTTP Get Welcome" through "Finish Matomo Setup", lines 45 through 156.
+
+- name: "WARNING: './runrole --reinstall matomo' CAN FAIL AS OF 2022-06-15, e.g. if /library/www/matomo already exists"
+  meta: noop
+
+# EXAMPLE OF ABOVE ERROR:
+
+# TASK [matomo : HTTP Get Welcome] ***************************************************************************************************************************************
+# fatal: [127.0.0.1]: FAILED! => {"cache_control": "private, no-cache, no-store", "changed": false, "connection": "close", "content_type": "text/html; charset=utf-8", "date": "Wed, 15 Jun 2022 05:07:41 GMT", "elapsed": 0, "expires": "Thu, 19 Nov 1981 08:52:00 GMT", "msg": "Status code was 500 and not [200]: HTTP Error 500: Internal Server Error", "pragma": "no-cache", "redirected": false, "server": "nginx/1.18.0 (Ubuntu)", "set_cookie": "MATOMO_SESSID=psak3aem27vrdrt8t2f016600f; path=/; HttpOnly; SameSite=Lax", "status": 500, "transfer_encoding": "chunked", "url": "http://box.lan/matomo/index.php?action=welcome", "x_matomo_request_id": "fbfd2"}
+
+- name: Start MariaDB
+  #action: service name=mysql state=started
+  systemd:
+    name: "{{ mysql_service }}"
+    state: started
+
+- name: Create MariaDB Database for Matomo
+  community.mysql.mysql_db:
+    name: "{{ matomo_db_name }}"
+    #login_unix_socket: /var/run/mysqld/mysqld.sock
+
+- name: Add Admin User to MariaDB Database
+  community.mysql.mysql_user:
+    name: "{{ matomo_db_user }}"
+    password: "{{ matomo_db_pass }}"
+    update_password: on_create    # OR SHOULD './runrole --reinstall matomo' FORCE A COMPLETELY CLEAN INSTALL?
+    priv: "{{ matomo_db_name }}.*:ALL"
+    #login_unix_socket: /var/run/mysqld/mysqld.sock
+
+- name: Download and Extract Matomo (~1 min)
+  unarchive:
+    src: "{{ matomo_dl_url }}"    # e.g. https://builds.matomo.org/matomo.tar.gz
+    dest: "{{ matomo_path }}"     # e.g. /library/www
+    remote_src: yes
+
+- name: Set Matomo Directory Permissions
+  file:
+    path: "{{ matomo_path }}/matomo"
+    recurse: yes
+    owner: "{{ apache_user }}"    # e.g. www-data
+    group: "{{ apache_user }}"
+
+- name: HTTP Get Welcome
+  uri:
+    url: "{{ matomo_full_url }}index.php?action=welcome"    # e.g. http://box.lan/matomo
+    method: GET
+    status_code: 200
+  register: matomo_welcome
+
+- debug:
+    var: matomo_welcome
+
+- name: Set a variable for the MATOMO_SESSID cookie
+  set_fact:
+    matomo_session_cookie: "MATOMO_SESSID={{ cookie.value }}"
+  when: cookie.key == "MATOMO_SESSID"
+  loop: "{{ matomo_welcome.cookies | dict2items }}"
+  loop_control:
+    loop_var: cookie
+
+- name: Get Matomo System Check
+  uri:
+    url: "{{ matomo_full_url }}index.php?action=systemCheck"
+    method: GET
+    headers:
+      Cookie: "{{ matomo_session_cookie }}"
+    return_content: true
+    timeout: 120
+    status_code: 200
+  register: matomo_system_check
+
+- debug:
+    var: matomo_system_check
+
+- name: Matomo Database Setup
+  uri:
+    url: "{{ matomo_full_url }}index.php?action=databaseSetup"
+    method: POST
+    headers:
+      Cookie: "{{ matomo_session_cookie }}"
+    body:
+      username: "{{ matomo_db_user }}"
+      password: "{{ matomo_db_pass }}"
+      dbname: "{{ matomo_db_name }}"
+      tables_prefix: "matomo_"
+      adapter: "PDO\\MYSQL"
+    body_format: form-urlencoded
+    status_code: 302
+  #register: matomo_database_setup
+
+- name: Matomo Table Creation
+  uri:
+    url: "{{ matomo_full_url }}index.php?action=tablesCreation&module=Installation"
+    method: GET
+    status_code: 200
+  register: matomo_table_creation
+
+- name: Set a variable for the MATOMO_SESSID cookie
+  set_fact:
+    matomo_session_cookie: "MATOMO_SESSID={{ cookie.value }}"
+  when:
+    - matomo_table_creation.cookies is defined
+    - matomo_table_creation.cookies | length > 0
+    - cookie.key == "MATOMO_SESSID"
+  loop: "{{ matomo_table_creation.cookies | dict2items }}"
+  loop_control:
+    loop_var: cookie
+
+- debug:
+    var: matomo_table_creation
+
+- name: Matomo User Setup
+  uri:
+    url: "{{ matomo_full_url }}index.php?action=setupSuperUser&module=Installation"
+    method: POST
+    headers:
+      Cookie: "{{ matomo_session_cookie }}"
+    body:
+      login: "{{ matomo_db_user }}"
+      password: "{{ matomo_db_pass }}"
+      password_bis: "{{ matomo_db_pass }}"
+      email: "nobody@dev.null"
+      subscribe_newsletter_piwikorg: 0
+      subscribe_newsletter_professionalservices: 0
+    body_format: form-urlencoded
+    status_code: 302
+  #register: matomo_setup_superuser
+
+- name: Configure Matomo to track IIAB
+  uri:
+    url: "{{ matomo_full_url }}index.php?action=firstWebsiteSetup&module=Installation"
+    method: POST
+    headers:
+      Cookie: "{{ matomo_session_cookie }}"
+    body:
+      siteName: "IIAB"
+      url: "{{ matomo_host_url }}"
+      ecommerce: 0
+    body_format: form-urlencoded
+    status_code: 302
+  #register: matomo_first_website_setup
+
+- name: Matomo Tracking Code
+  uri:
+    url: "{{ matomo_full_url }}index.php?action=trackingCode&module=Installation&site_idSite=1&site_name={{ matomo_host_url }}"
+    method: GET
+    headers:
+      Cookie: "{{ matomo_session_cookie }}"
+    return_content: true
+    status_code: 200
+  #register: matomo_tracking_code
+
+- name: Finish Matomo Setup
+  uri:
+    url: "{{ matomo_full_url }}index.php?action=finished&module=Installation"
+    method: POST
+    headers:
+      Cookie: "{{ matomo_session_cookie }}"
+    body:
+      do_not_track: 1
+      anonymise_ip: 1
+      submit: "Continue to Matomo"
+    body_format: form-urlencoded
+    status_code: 302
+
+- name: Start Collecting Matomo Data
+  cron:
+    name: "MatomoDataIngestionOnReboot"
+    special_time: reboot
+    job: "{{ matomo_cronjob }}"
+    user: root
+    cron_file: "matomo_reboot"
+
+- name: Run Daily Job Collecting Matomo Data
+  cron:
+    name: "DailyMatomoDataIngestion"
+    minute: "0"
+    hour: "0"
+    job: "{{ matomo_cronjob }}"
+    user: root
+    cron_file: "matomo_daily"
+
+
+# RECORD Matomo AS INSTALLED
+
+- name: "Set 'matomo_installed: True'"
+  set_fact:
+    matomo_installed: True
+
+- name: "Add 'matomo_installed: True' to {{ iiab_state_file }}"
+  lineinfile:
+    path: "{{ iiab_state_file }}"    # /etc/iiab/iiab_state.yml
+    regexp: '^matomo_installed'
+    line: 'matomo_installed: True'

roles/matomo/tasks/main.yml

@@ -0,0 +1,55 @@
+# "How do i fail a task in Ansible if the variable contains a boolean value?
+# I want to perform input validation for Ansible playbooks"
+# https://stackoverflow.com/questions/46664127/how-do-i-fail-a-task-in-ansible-if-the-variable-contains-a-boolean-value-i-want/46667499#46667499
+
+# We assume 0-init/tasks/validate_vars.yml has DEFINITELY been run, so no need
+# to re-check whether vars are defined here.  As Ansible vars cannot be unset:
+# https://serverfault.com/questions/856729/how-to-destroy-delete-unset-a-variable-value-in-ansible
+
+- name: Assert that "matomo_install is sameas true" (boolean not string etc)
+  assert:
+    that: matomo_install is sameas true
+    fail_msg: "PLEASE SET 'matomo_install: True' e.g. IN: /etc/iiab/local_vars.yml"
+    quiet: yes
+
+- name: Assert that "matomo_enabled | type_debug == 'bool'" (boolean not string etc)
+  assert:
+    that: matomo_enabled | type_debug == 'bool'
+    fail_msg: "PLEASE GIVE VARIABLE 'matomo_enabled' A PROPER (UNQUOTED) ANSIBLE BOOLEAN VALUE e.g. IN: /etc/iiab/local_vars.yml"
+    quiet: yes
+
+
+- block:
+
+  - name: Enable/Disable/Reload NGINX for OSM, if nginx_enabled
+    include_tasks: nginx.yml
+
+  - name: Install Matomo if 'matomo_installed' not defined, e.g. in {{ iiab_state_file }}    # /etc/iiab/iiab_state.yml
+    include_tasks: install.yml
+    when: matomo_installed is undefined
+
+  # LET'S ADD THIS "ON/OFF SWITCH" IF POSS!
+  # - include_tasks: enable-or-disable.yml
+
+  - name: Add 'matomo' variable values to {{ iiab_ini_file }}
+    ini_file:
+      path: "{{ iiab_ini_file }}"    # /etc/iiab/iiab.ini
+      section: matomo
+      option: "{{ item.option }}"
+      value: "{{ item.value | string }}"
+    with_items:
+      - option: name
+        value: Matomo
+      - option: description
+        value: '"Matomo is a web analytics alternative to Google Analytics, emphasizing privacy and data ownership."'
+      - option: matomo_install
+        value: "{{ matomo_install }}"
+      - option: matomo_enabled
+        value: "{{ matomo_enabled }}"
+      
+  rescue:
+
+  - name: 'SEE ERROR ABOVE (skip_role_on_error: {{ skip_role_on_error }})'
+    fail:
+      msg: ""
+    when: not skip_role_on_error

roles/matomo/tasks/nginx.yml

@@ -0,0 +1,16 @@
+- name: Enable http://box/maps & http://box/matomo via NGINX, by installing {{ nginx_conf_dir }}/matomo-nginx.conf from template
+  template:
+    src: matomo-nginx.conf.j2
+    dest: "{{ nginx_conf_dir }}/matomo-nginx.conf"    # /etc/nginx/conf.d
+  when: matomo_enabled
+
+- name: Disable http://box/maps & http://box/matomo via NGINX, by removing {{ nginx_conf_dir }}/matomo-nginx.conf
+  file:
+    path: "{{ nginx_conf_dir }}/matomo-nginx.conf"    # /etc/nginx/conf.d
+    state: absent
+  when: not matomo_enabled
+
+- name: Reload 'nginx' systemd service
+  systemd:
+    name: nginx
+    state: reloaded

roles/matomo/templates/matomo-nginx.conf.j2

@@ -0,0 +1,17 @@
+location ~ ^/matomo(.*)\.php(.*)$ {
+    alias /library/www/matomo$1.php$2;    # /library/www/matomo
+    proxy_set_header X-Real-IP  $remote_addr;
+    proxy_set_header X-Forwarded-For $remote_addr;
+    proxy_set_header Host $host;
+    fastcgi_pass php;
+    fastcgi_index index.html;
+    include fastcgi_params;
+    fastcgi_split_path_info ^(.+\.php)(.*)$;
+    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
+    fastcgi_param SCRIPT_NAME     $fastcgi_script_name;
+    fastcgi_param PATH_INFO       $2;
+}
+
+location ~ ^/matomo/ {
+    root /library/www;
+}

vars/default_vars.yml

@@ -573,6 +573,10 @@ transmission_kalite_languages:
 awstats_install: True
 awstats_enabled: True
 
+# Matomo is a web analytics alternative to Google Analytics, emphasizing privacy and data ownership.
+matomo_install: True
+matomo_enabled: True
+
 # Process supervision tool - from https://mmonit.com/monit/
 # 2020-09-22 WARNING: both vars are IGNORED on Debian 10 due to: iiab/iiab#1849
 monit_install: False

vars/local_vars_large.yml

@@ -343,6 +343,10 @@ transmission_kalite_languages:
 awstats_install: True
 awstats_enabled: True
 
+# Matomo is a web analytics alternative to Google Analytics, emphasizing privacy and data ownership.
+matomo_install: True
+matomo_enabled: True
+
 # Process supervision tool - from https://mmonit.com/monit/
 # 2020-09-22 WARNING: both vars are IGNORED on Debian 10 due to: iiab/iiab#1849
 monit_install: False

vars/local_vars_medium.yml

@@ -343,6 +343,10 @@ transmission_kalite_languages:
 awstats_install: True
 awstats_enabled: True
 
+# Matomo is a web analytics alternative to Google Analytics, emphasizing privacy and data ownership.
+matomo_install: True
+matomo_enabled: True
+
 # Process supervision tool - from https://mmonit.com/monit/
 # 2020-09-22 WARNING: both vars are IGNORED on Debian 10 due to: iiab/iiab#1849
 monit_install: False

vars/local_vars_small.yml

@@ -343,6 +343,10 @@ transmission_kalite_languages:
 awstats_install: True
 awstats_enabled: True
 
+# Matomo is a web analytics alternative to Google Analytics, emphasizing privacy and data ownership.
+matomo_install: True
+matomo_enabled: True
+
 # Process supervision tool - from https://mmonit.com/monit/
 # 2020-09-22 WARNING: both vars are IGNORED on Debian 10 due to: iiab/iiab#1849
 monit_install: False

vars/local_vars_unittest.yml

@@ -343,6 +343,10 @@ transmission_kalite_languages:
 awstats_install: False
 awstats_enabled: False
 
+# Matomo is a web analytics alternative to Google Analytics, emphasizing privacy and data ownership.
+matomo_install: False
+matomo_enabled: False
+
 # Process supervision tool - from https://mmonit.com/monit/
 # 2020-09-22 WARNING: both vars are IGNORED on Debian 10 due to: iiab/iiab#1849
 monit_install: False