Monday, September 19, 2016

NodeMCU(ESP8266) as provider of NTP date and time over serial port

I've been looking for a date and time source for my Arduino projects. Arduino has limited resources, and I would like to use them for the actual work. So I've decided to use NodeMCU with Lua to access date and time from the internet, format it in a way that meets my requirements, and provide such results over the serial port.

There are many ways to access date and time from the internet, but mainly those interfaces change over time, and code using them must be adopted. I was looking for something that would last for years without interaction from my side. That's why I've decided to use Network Time Protocol. There is only one catch: it's UTC time without a date. It was not such a big deal because there are a lot of algorithms out there, so I've just adopted one of them.

Now it's possible to obtain local date and time from NodeMCU over the serial port, and the whole thing is reliable and will last for many years without touching it :)

Here is the source code: https://github.com/maciejmiklas/NodeMCUUtils

The implementation is divided into small building blocks; putting them together will give you the desired serial API. This is just a short overview:

  • Date Format - calculates local date and time based on the timestamp
  • WiFi Access  - simple facade for connecting to WiFi
  • NTP Time - obtains UTC timestamp form given NTP server
  • NTP Clock - keeps actual UTC timestamp and synchronizes it periodically with the NTP server
  • Serial API - finally, this one provides API for date and time


Date Format

Provides functionality to get local date and time from timestamp given in seconds since 1970.01.01

For such code:
collectgarbage() print("heap before", node.heap())

require "dateformat"
require "dateformatEurope"
local ts = 1463145687
df.setEuropeTime(ts, 3600) -- function requires GMT offset for your city

print(string.format("%04u-%02u-%02u %02u:%02u:%02d", 
    df.year, df.month, df.day, df.hour, df.min, df.sec))
print("DayOfWeek: ", df.dayOfWeek)

collectgarbage() print("heap after", node.heap())

You will get this output:
heap before 44704
2016-05-13 15:21:27
DayOfWeek:  6
heap after  39280

To format the date for the USA, you have to replace "dateformatEurope" with "dateformatAmerica" and call setAmericaTime instead of setEuropeTime. Functionality is divided into small scripts to save some RAM.


WiFi Access

It's a simple facade for connecting to WiFi. You have to provide connection credentials and a function that will be executed after the connection has been established.

execute(...) connects to WiFi, which can take some time. You can still call this method multiple times. In such cases, callbacks will be stored in the queue and executed after establishing a WiFi connection.

require "wlan"

wlan.debug = true

local function printAbc() 
    print("ABC")
end

wlan.setup("free wlan", "12345678")
wlan.execute(printAbc)

Configuring WiFi on:   free wlan
status  1
status  1
status  5
Got WiFi connection:   172.20.10.6 255.255.255.240 172.20.10.1
ABC


NTP Time

This simple facade connects to a given NTP server, requests UTC time from it, and once a response has been received, it calls the given callback function. 

The example below executes the following chain: WiFi -> NTP -> Date Format. 
So in the first step, we create a WLAN connection and register a callback function that will be executed after the connection has been established. This callback function requests time from the NTP server (ntp.requestTime). 
On the ntp object, we are registering another function that will get called after the NTP response has been received: printTime(ts).

collectgarbage() print("RAM init", node.heap())

require "wlan"
require "ntp"
require "dateformatEurope";

collectgarbage() print("RAM after require", node.heap())

ntp = NtpFactory:fromDefaultServer():withDebug()
wlan.debug = true

local function printTime(ts) 
    collectgarbage() print("RAM before printTime", node.heap())
    
    df.setEuropeTime(ts, 3600)
    
    print("NTP Local Time:", string.format("%04u-%02u-%02u %02u:%02u:%02d", 
        df.year, df.month, df.day, df.hour, df.min, df.sec))
    print("Summer Time:", df.summerTime)
    print("Day of Week:", df.dayOfWeek)
    
    collectgarbage() print("RAM after printTime", node.heap())
end

ntp:registerResponseCallback(printTime)

wlan.setup("free wlan", "12345678")
wlan.execute(function() ntp:requestTime() end)

collectgarbage() print("RAM callbacks", node.heap())

and console output:
RAM init    43328
RAM after require   30920
Configuring WiFi on:   free wlan
RAM callbacks   30688
status  1
status  1
status  5
Got WiFi connection:   172.20.10.6 255.255.255.240 172.20.10.1
NTP request:    pool.ntp.org
NTP request:    194.29.130.252
NTP response:   11:59:34
RAM before printTime    31120
NTP Local Time: 2016-07-12 13:59:34
Summer Time: 
Day of Week:    3
RAM after printTime 30928


NTP Clock

This script provides functionality to run a clock with the precision of one second and to synchronize this clock every few hours with the NTP server. 

In the code below, we first configure WiFi access. Once the WiFi access has been established, it will call ntpc.start(). This function will start a clock synchronizing with a given NTP server every minute. Now you can access actual UTC time in seconds over ntpc.current. To show that it's working, we have registered a timer that will call: printTime() every second. This function reads current time as ntpc.current and prints it as local time. 

collectgarbage() print("RAM init", node.heap())

require "dateformatEurope";
require "ntpClock";
require "wlan";

collectgarbage() print("RAM after require", node.heap())

ntpc.debug = true
wlan.debug = true

wlan.setup("free wlan", "12345678")
wlan.execute(function() ntpc.start("pool.ntp.org", 60) end)

local function printTime() 
    collectgarbage() print("RAM in printTime", node.heap())
    
    df.setEuropeTime(ntpc.current, 3600)
    
    print("Time:", string.format("%04u-%02u-%02u %02u:%02u:%02d", 
        df.year, df.month, df.day, df.hour, df.min, df.sec))
    print("Summer Time:", df.summerTime)
    print("Day of Week:", df.dayOfWeek)
end

tmr.alarm(2, 30000, tmr.ALARM_AUTO, printTime)

so this is the output:
RAM init    43784
RAM after require   29408
Configuring WiFi on:    free wlan
status  1
status  5
Got WiFi connection:    192.168.2.113   255.255.255.0   192.168.2.1

NTP request:    pool.ntp.org
NTP request:    195.50.171.101
NTP response:   17:09:46

RAM in printTime    29664
Time:   2016-08-08 19:10:08
Summer Time:    true
Day of Week:    2

RAM in printTime    29808
Time:   2016-08-08 19:10:38
Summer Time:    true
Day of Week:    2

NTP request:    pool.ntp.org
NTP request:    195.50.171.101
NTP response:   17:10:46

RAM in printTime    29680
Time:   2016-08-08 19:11:08
Summer Time:    true
Day of Week:    2

RAM in printTime    29808
Time:   2016-08-08 19:11:38
Summer Time:    true
Day of Week:    2

NTP request:    pool.ntp.org
NTP request:    131.188.3.221
NTP response:   17:11:46

RAM in printTime    29680
Time:   2016-08-08 19:12:08
Summer Time:    true
Day of Week:    2

RAM in printTime    29808
Time:   2016-08-08 19:12:38
Summer Time:    true
Day of Week:    2


Serial API

Serial API exposes a simple interface that provides access to diagnostic info and date to be accessed outside NodeMCU - for example, by Arduino.

Serial API is divided into a few Lua scripts. Loading of each script will automatically add new API commands:
- serialAPI.lua - has to be always loaded. It initializes the serial interface with a few diagnostics commands.
- serialAPIClock.lua - access to clock including date formatter.

Each script above registers a set of commands as keys of scmd table and contains further documentation.

The example below provides date access over the serial port:

require "credentials"
require "serialAPI"
require "serialAPIClock"

ntpc.syncPeriodSec = 900 -- 15 min
sapi.baud = 115200

-- setup wlan required by NTP clock
wlan.setup("free wlan", "12345678")

-- start serial API by enabling gpio and uart
sapi.start()

-- start NTP synchronization
ntpc.start("pool.ntp.org")

Here are few Serial API commands and their responses:
# free ram
>GFR
10664

# WiFi status
>GWS
5

# date and time (24h) in format: yyyy-mm-dd HHLmm:ss
>CF1
2016-09-16 10:45:25

# date in format: yyyy-mm-dd
>CH2
10:45:59

Firmware

Executing multiple scripts can lead to out-of-memory issues on NodeMCU. One possibility to solve it is to build custom firmware containing only a minimal set of node-mcu modules: cjson, file, gpio, net, node, tmr, uart, and WiFi. This blog provides a detailed upgrade procedure: http://maciej-miklas.blogspot.de/2016/08/installing-nodemcu-v15-on-eps8266-esp.html