From Stu2
Revision as of 13:26, 24 December 2019 by Stu2 (talk | contribs) (Manual Programming)
Jump to navigation Jump to search

ESP32 Programming. The 'new' ESP32 dev boards are preprogrammed with a LUA script. To get an Arduino to load, press and hold the 'Boot' button while uploading the Arduino code. Hold the boot button until finished loading the code.



Some Planning:

  • mosquitto - can use base image. No customization really required. If moved to cloud, may want to figure out TLS and add passwords, etc.
  • node-red - customize Dockerfile to load modules. Update flows manually.
  • influxdb - need to store data in external volume
  • grafana
  • docker - create bridge, try docker compose to bundle all this stuff
  • cloud - cloud is expensive, but we could move mosquitto, influx and grafana to cloud, use flow in node red to push updates

Cheat sheet

docker ps -a    -lists all containers
docker stop [name]
docker container prune  -remove all containers in exit status
docker container rm [name]
docker images
docker rm (name) - get rid of a container
docker-compose up
docker-compose up -d -> run in background
docker-compose logs --tail
docker exec -u 0 -it [DOCKERID] bash -> get a shell inside a container

Setup the network.

docker network create wwi-net

Interesting to note that we can use the docker names on this network. e.g. the MQTT server can be mqtt.

Individual Container Build Notes

The following sections have notes about how to build each container. No need to do this with docker-compose, except for node-red


Some mosquitto help.

mkdir ~/mosquitto/config
mkdir ~/mosquitto/data
mkdir ~/mosquitto/log

Edit ~/mosquitto/config/mosquitto.conf

 persistence true
 persistence_location /mosquitto/data/
 log_dest file /mosquitto/log/mosquitto.log
 allow_anonymous true

docker run --name mqtt -d -p 1883:1883 --net wwi-net -v $(pwd)/mosquitto/config/mosquitto.conf:/mosquitto/config/mosquitto.conf -v $(pwd)/mosquitto/data:/mosquitto/data -v $(pwd)/mosquitto/log:/mosquitto/log eclipse-mosquitto

mosquitto_pub -h -m "test" -t test/switch -d

https://blog.container-solutions.com/understanding-volumes-docker - notes on volumes. The eclipse-mosquitto readme should be more explicit about how to use these volumes. 


git node-red-docker as per this: https://nodered.org/docs/getting-started/docker
Create Dockfile in ~/node-red-docker/v1.0/Dockerfile. I wish I could use Alpine (:slim) :latest is 900MB. Dockerfile:

FROM nodered/node-red-docker:latest


RUN npm install node-red-dashboard
RUN npm install node-red-contrib-influxdb

Then build image:

docker build -f v1.0/Dockerfile -t rpt-nodered:1.0 .

Run to build container:
docker run -d --net wwi-net --name rpt-nodered -p 1880:1880 rpt-nodered:1.0

Once built, start and stop as usual. (docker stop rpt-nodered, docker start rpt-nodered)

Import flows into container via node-red development dashboard. I think these are stored in the container. We'll see when I move the container.


Some influxdb help:

Create directory for data. e.g. ~/influxdb
Pull out the configuration file to the present working directory (PWD) using:

docker run --rm influxdb influxd config > influxdb.conf

Modify config file, if needed.

Create container and run, which uses ~/influxdb.conf and stores data in ~/influxdb

docker run --name rpt-influxdb --net wwi-net -d -p 8086:8086 -v $(pwd)/influxdb/influxdb:/var/lib/influxdb -v $(pwd)/influxdb.conf:/etc/influxdb/influxdb.conf:ro influxdb -config /etc/influxdb/influxdb.conf

Create measurement database:

docker exec -it rpt-influxdb influx

create database meters


Some softether help:

Create config directory ~/softether/config and log directory ~/softether/log

docker run -d --cap-add NET_ADMIN --name softether-vpn-server -p 443:443/tcp -p 992:992/tcp -p 1194:1194/udp -p 5555:5555/tcp -v $(pwd)/softether/config:/etc/vpnserver:Z -v $(pwd)/softether/logs:/var/log/vpnserver:Z amary/softether-vpn-server

Note cap-add is a linux capability for the network.

SELINUX label - the Z option indicates that the bind mount content is private and unshared.

Manage via softether server management gui.


Reset admin password

docker exec -u 0 -it rpt-grafana bash

ln -s /var/lib/grafana  /usr/share/grafana/data
ln -s /var/log/grafana /usr/share/grafana/data/logs

grafana-cli --homepath "/usr/share/grafana" admin reset-admin-password admin

Docker Compose

After I figured out how to create stand alone containers, I figured out docker-composer.

Do the following:

  • Copy dc/docker-compose.yml to new computer - this is the main configuration file!
  • Create .env in dc directory, add the secret stuff
  • Copy all the data directories over to the new computer. (grafana,influxdb,node-red-data,node-red-docker,mosquitto)
  • Build node-red-docker, first. See the node-red section above. This creates rpt-nodeRed image we need, which has the influxdb and dashboard modules
  • add user grafana and change userid and groupid to 472
  • fix permission issues. (probably mosquitto)
  • install npm and node-red-admin and run node-red-admin hash-pw. Put the hash in /node-red-data/settings.js - this sets up an admin password
  • cd into dc directory and run 'docker-compose up'. Look through the output for errors.
  • docker-compose up -d then fix it so this runs on startup.

I bet I can automate all that stuff above, but will figure it out later.

Docker Compose Starup

Use 'restart: always' in docker-compose.yml.

Enable the docker service

sudo systemctl enable docker
docker-compose up -d  -> run once

Manual Programming

I switched to platform IO and vscode. Works great. However, the PIO gateway was down, so I needed to figure out how to program the ESP32 manually from the results of building under PIO. Took several hours to track down the info. I found the esptool command, by running 'pio run upload -v' on the command line in the vscode terminal. I used SCP to move the firmware files from my linux workstation the repeater computer. There are four files required. I found them in the esptool command line as follows:

--chip esp32 --port "/dev/ttyUSB4" --baud 460800 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 40m --flash_size detect 0x1000 /home/stu/.platformio/packages/framework-arduinoespressif32/tools/sdk/bin/bootloader_dio_40m.bin 0x8000 /data/home/stu/Documents/PlatformIO/Projects/PowerMeter/.pio/build/esp32doit-devkit-v1/partitions.bin 0xe000 /home/stu/.platformio/packages/framework-arduinoespressif32/tools/partitions/boot_app0.bin 0x10000 .pio/build/esp32doit-devkit-v1/firmware.bin

Once the files are located on the repeater computer, I can program the ESP32 as follows. Note that both boot loader files don't change. They are specific to the ESP32.

For PowerMeter:

esptool --chip esp32 --port "/dev/ttyUSB0" --baud 460800 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 40m --flash_size detect 0x1000 /home/stu/pio/bootloader_dio_40m.bin 0x8000 /home/stu/powermeter/partitions.bin 0xe000 /home/stu/pio/boot_app0.bin 0x10000 /home/stu/powermeter/firmware.bin

For PowerMeterSetup
esptool --chip esp32 --port "/dev/ttyUSB0" --baud 460800 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 40m --flash_size detect 0x1000 /home/stu/pio/bootloader_dio_40m.bin 0x8000 /home/stu/powermeterSetUp/partitions.bin 0xe000 /home/stu/pio/boot_app0.bin 0x10000 /home/stu/powermeterSetUp/firmware.bin

For some reason, reading the eeprom using EEPROM.read(0); doesn't work anymore. So I hard coded the model #, recompiled, scp'ed and programmed each power meter by hand. Then, avahi handed out the wrong IP address, so I hard coded the MQTT broker in the code. Hopefully, I can figure this one out someday!