Bottom-up SOAP Services with Apache CXF and Custom XSD

SOAP is not very popular these days, primarily because of almost nonhuman readable WSDL and protocol that puts a lot of overhead into a simple message. It's tough to send a message using a browser or curl. Not mentioning the fact that we have to obtain WSDL somehow.

But on the other hand side, SOAP has some advantages over REST. It allows us to define a contract and specify the data it will process. Once you have such WSLD, you can throw it over the hedge, and you do not have to answer questions like how do I delete a created resource, what 403 means in this case, or is this call idempotent?
WSLD contains everything needed to define an RPC interface and document it. I would choose REST for most cases, but still, there are some good use cases for SOAP.

This one thing bothers me about SOAP: how do I get proper WSLD? Should I use a tool to generate it or write it? Honestly... I really do not want to dig into SOAP to just write WSLD, and I do not want to spend hours playing around with some tools generating WSLD, importing it into my project, and binding generated stuff to actual implementation only to find out about the end that it does not operate as expected.

The perfect solution would be to write Java code, annotate it, and generate WSDL out of it. Those annotations should provide enough flexibility to influence WSLD generation so that we archive something that can be delivered to the customers.

I've tried a few Java frameworks, and all provide a way to generate WSLD out of Java code, but they all need to include one fundamental feature: you cannot offer XSD for data types. So it's impossible to specify the content of an email field or provide a format for a date. We get only half of a possible functionality of a WSDL: method calls, exception handling, documentation, and security, but there is no way to specify the data format.
I would like to have WSLD that is as precise as possible so that the client can call the particular method and know precisely what is possible and is not allowed.

Let's start with the standard SOAP Service generated out of Java code. We will use Apache CXF and Spring Boot for it. The source code is here: https://github.com/maciejmiklas/apache-cxf-soap.

It's a simple application where users can submit registration as a SOAP Request on http://localhost:8080/soap/Registration:


This code outpost following WSDL:
As you can see, this WSDL has little info about data types (lines 9-22). The email is mandatory, but there is no way to provide any further assertions.
The problem lies in JAXB. It ignores annotations to influence generated XML schema. There is a pull request (jaxb-facets) that would solve this issue, but it's already a few years old and looks like it will only be integrated for a while.

The limitation is not caused by SOAP frameworks but by the fact that they are based on JAXB.

This does not change the fact that I will not write WSLD myself. There has to be a better way!

We will implement a simple extension for Apache CXF to combine generated WSLD with custom XSD. We have to write XSD in this case, but CXF will generate WSLD for us and include the given schema.
XSD is mighty, so I prefer writing it by hand because it's possible to specify the data format precisely. Using annotations to generate XSD would be convenient, but it would still cover only some common areas.
The implementation below has its limitations. You might run into some issues. It might stop working after the next CXF update. But! I use it, and it does what I need, so it might be something for you as well ;)

Now we are going to modify the first example. The idea is to write XSD that defines simple types, references those types in transfer classes, and generates WSLD, combining them all.
In the beginning, we have to write a schema that defines types for transfer objects: The original source code will be modified only in a few places:

• We will replace the default Apache CXF data binding with a custom implementation. It reads Schama from the file and integrates it into generated WSLD,
• Some fields in transfer objects are annotated with @XmlSchemaType - this annotation provides a connection between Java types and types defined in XSD. For example, the ExRegistration#email is annotated with @XmlSchemaType(name = "email"), and XSD contains the email type. The email generated in WSDL references the type from a provided schema,
• the classes following the pattern *Registration* have been renamed to *ExRegistration*

Here is our final WSDL:

Spring Beans Discovery through Java Config in Practice

Spring Java Config is a great tool that gives us a precise way to control an application's bootstrap and wiring. However, it only sometimes works as expected. We occasionally run into strange issues where a particular bean is not instantiated at the expected time, lazy loading does not work as expected, or the whole configuration is ignored for no apparent reason.

To get a better understanding of the inner workings of Spring bean discovery, I created some examples that can be found here: https://github.com/maciejmiklas/spring-context-test.git

Each example has a name: exa (example A) or exb (example B). Each example has several exercises, like exa has: exa01, exa02, ..., ex05.

Bean Loading Order - Single Config

https://github.com/maciejmiklas/spring-context-test/tree/master/src/main/java/org/test/context/exa

exa01

This is the first exercise. It defines a basic structure that will be modified as we go along.
The main method loads Spring context from the configuration file Conf.java. This configuration instructs Spring to create 3 beans. Each of these beans has two log statements, the first in the constructor and the second in @PostConstruct. The console output after the execution of this example is shown below the code.

Nothing spectacular happens here: beans are initialized from A to C right after the constructor Spring executes also @PostConstruct. There are no dependencies between the beans.

exa02

Let's modify the order of factory methods in Conf.java.

The bean initialization order has changed. It means that method names do not matter, but the physical order within a class. It can change with JVM, so you should not rely on it!

exa03

Now we will extract the definition of BeanB into a dedicated configuration file:

Spring loads bean definitions from the imported configuration in the first place.

exa04

We will modify the previous exercise so that BeanB depends on BeanA. Spring loads dependent beans in the first place. It does not matter where it's located. Spring will scan all possible configurations to find it.

exa05

The same as exa04, but BeanB does not have @DependsOn("beanA"). Instead, BeanB injects BeanA. We have a similar dependency situation to exa4: BanB depends on BeanA. However @PostConstructs are called in a different order: Spring creates an instance of BeanB, and during the construction phase (constructor), references to BeanA are not set. Once the instance of BeanB is created, BeanA will be created and injected into BeanB. BeanB cannot access BeanA in the constructor, first after a complete initialization is done - in the method @PostConstruct.
Spring had to modify the order of @PostConstruct calls to ensure that the bean references were not null during an initialization phase.

Summary

• within a single configuration class, the bean instantiation order depends only on the methods order within this class, not method names. However, this might depend on JVM,
• Spring loads, in the first place, imported bean definitions. The previous rule applies only to the situation where direct bean definitions do not depend on imported beans. If this is the case, Spring will load the dependent bean in the first place,
• the initialization code has to be placed in the @PostConstruct method and not in a constructor. Otherwise, references to some beans might be null.

Bean Loading Order - Mixed Config

https://github.com/maciejmiklas/spring-context-test/tree/master/src/main/java/org/test/context/exb

exb01

BeanA, BeanB, and BeanC are created through dedicated configuration classes: ConfBeanA, ConfBeanB, and ConfBeanC. BeanD is declared in Conf Beans are created in order: A, B, C, D.

exb02

Similar to exb01, but the configuration class ConfBeanB has been renamed to ConfBeanXB.
The instantiation order remains unchanged.

exb03

The same as exb01, but the factory method in ConfBeanB has been renamed from beanB to xyz. The instantiation order remains unchanged.

exb04

The same as exb01, but BeanB has been renamed to BeanXB. This change did not influence instantiation order.

exb05

Similar to exb01, but we've changed the import order from A, B, C into B, A, C. The instantiation order has changed as well.

exb06

Same as exb01, but BeanC has been injected into BeanA. The instantiation order has been changed because BeanC had to be created before BeanA

Summary

• when the @Import statement contains multiple classes, Spring will load them from the left to the right side - it's iterating over an array, and names of configuration classes do not matter in this case,
• within single configuration class physical order of methods matters, not their names.

Duplicated Bean Name

exc01

We have 3 beans, A, B, and C, with dedicated config here. BeanA injects BeanB and BeanC.

exc02

The same as exc01, but we've renamed factory method ConfBeanB#beanB() into ConfBeanB#beanC(). Spring goes over our @Import declaration to determine all possible bean definitions: ConfBeanA, ConfBeanB, ConfBeanC. There are two factory methods with the same name: ConfBeanB#beanC() and ConfBeanC#beanC(), so ConfBeanC overwrites the bean created through ConfBeanB because it creates a bean with the same name.

exc03

Similar to exc02, but this time we've renamed method ConfBeanC#beanC() into ConfBeanC#beanB(), so that we have two methods beanB, and not as it was in exc02 two methods with the name: beanC. The output is still the same, BeanB is missing. We've just changed the name of the factory method, and we already know that names do not change loading order, and we still have two beans with the same name.

exc04

Similar to exc02, but we've changed the import order for config classes from A, B, C to A, C, B. Now ConfBeanB will get scanned on end, and it overwrites the previous bean with the same name, so BeanC is missing.

exc05 

It's a modified exc02. We've set AllowBeanDefinitionOverriding to false on Application Context. Bean overwriting is disabled now, so instead of bean not found, we are getting an exception that we are trying to register two beans under the same name.

exc06

The same as exc02, there are still two factory methods: beanC. Additionally, BeanB and BeanC are implementing the common interface. Now BeanA does not inject BeanB and BeanC directly but injects a collection of beans implementing our interface. In the case of direct injection, one of the beans was missing; now both are there!

Summary

In the case of two beans with the same name, the last one wins. You cannot directly inject such overwritten beans, but you can inject a collection of such beans sharing a common interface.

Lazy Loading

exd01

We have three beans: A, B, and C, and there are no dependencies between them. BeanC is being defined through a dedicated configuration class. Nothing special here. The initialization order is as expected.

exd02

The same as exd02, but we've annotated BeanB as lazy. Spring does not load BeanB at all. There are no dependencies to that bean.

exd03

Now we've injected BeanB into BeanA. @Lazy is set on injection point and bean definition. Additionally, we have defined a method on BeanA that calls a method on BeanB: ApplicationExD03 -> beanA.method() -> beanB.method() Spring postpones the creation of BeanB until the method is being called on it, @Lazy works as expected.

exd04

Same as exd03, but @Lazy has been removed on the injection point. BeanB is not lazy anymore, @Lazy on bean definition is not enough.

exd05

Same as exd03, but @Lazy has been removed from the bean configuration. BeanB is not lazy anymore, @Lazy on bean definition is not enough.

Summary

• lazy annotation has to be provided on configuration, and all injection points, otherwise Spring eagerly initializes such beans,
• lazy beans that are not referenced (injected) are not loaded at all,
• you should not rely on @Lazy unless you are 100% sure you can control all possible injection points. Otherwise, one missing Lazy-Annotation will disable laziness on the such bean.

Thermostat based on Arduino

We will build a Thermostat based on Arduino, a temperature sensor, and a relay this time.
You can find it on GitHub

This Thermostat gives you the possibility to drive multiple devices in order to control temperature. In my case, I have installed two fans in my attic in order to cool it down in summer. There are no windows, so I had to force airflow. The first fan starts when the temperature reaches 35 degrees, and the second over 45.

You can control any reasonable amount of units, it's all configurable. You have also had access to basic statistics for the past two weeks:

Runtime of whole system

ON time for each relay

Statistics for 14 days

Statistics for 7th day

Configuration

The whole configuration is stored in Config.h. You can change PINs controlling relays, buttons, and input for reading temperature, thresholds, or timings.

Hardware

Buttons

Two buttons are dedicated for menu navigation and the third one resets statistics. They are stored in EEPROM (Storage.cpp) once a day. Pins assigned to buttons can be found in Config.h

Configuring Relays

Let's assume that we would like to have 3 relays:

ID:0, PIN: 1, Temperature setpoint: 20

ID:1, PIN: 10, Temperature setpoint: 30

ID:2, PIN: 11, Temperature setpoint: 40

First, you have to make sure that PIN of your choice is not already taken. All pins can be found in Config.h, they are defined by variables starting with DIG_PIN.

You have to edit Config.h and configure PINs, thresholds and amount of relays. Obviously, some properties already exist, so you have to just edit them.

const static uint8_t DIG_PIN_RELAY_0 = 1;
const static uint8_t DIG_PIN_RELAY_1 = 10;
const static uint8_t DIG_PIN_RELAY_2 = 11;

const static uint8_t RELAYS_AMOUNT = 3;

const static int16_t RELAY_TEMP_SET_POINT_0 = 20;
const static int16_t RELAY_TEMP_SET_POINT_1 = 30;
const static int16_t RELAY_TEMP_SET_POINT_2 = 40;

Now we have to set up relays and controller, this happens in RelayDriver.cpp

    initRelayHysteresisController(0, DIG_PIN_RELAY_0, RELAY_TEMP_SET_POINT_0);
    initRelayHysteresisController(1, DIG_PIN_RELAY_1, RELAY_TEMP_SET_POINT_1);
    initRelayHysteresisController(2, DIG_PIN_RELAY_2, RELAY_TEMP_SET_POINT_2);

Choosing Controller

There two controllers available Hysteresis and PID

Hysteresis Controller

It's the one chosen in the example above, it has few additional configurations:

const static uint32_t RELAY_DELAY_AFTER_SWITCH_MS = 300000; // 5 minutes
const static uint32_t RHC_RELAY_MIN_SWITCH_MS = 3600000;

RELAY_DELAY_AFTER_SWITCH_MS gives a wait time for switching to the next relay. Imagine that configuration from our example would start working in 40 degrees environment. This would result in enabling all three relays at the same time. This could eventually lead to high power consumption - depending on what you are controlling, an electric engine, for example, consumes more power during the start. In our case, switching relays have the following flow: the first relay goes, waits 5 minutes, the second goes on, waits 5 minutes, third goes on.

RHC_RELAY_MIN_SWITCH_MS defines hysteresis, it's the minimum frequency for a particular relay to change its state. Once it's on, it will remain on for at least this period of time, ignoring temperature changes. This is quite useful if you control electric motors since each switch negatively impacts live time.

PID Controller

This is an advanced topic. Implementing such a controller is a simple task, finding the right amplitude settings is a different story. 

To use the PID controller, you must change initRelayHysteresisController(.....) to initRelayPiDController(....) and find the right settings for it. You will find them in Config.h 

I've implemented a simple simulator in Java so that it's possible to visualize the results. It can be found in the folder: pidsimulator.

Below you can see simulations for two controllers PID, and  P. PID is not perfectly stable because I did not apply any sophisticated algorithm to find the right values.

On both plots required temperature is set to 30 (blue). Current temperature indicates read line. The relay has two states ON and OFF. When it's enabled, temperature drops by 1.5, when it's disabled, it rises by 0.5.

Software Design

Message Bus

Different software modules have to communicate with each other, hopefully not both ways ;) 

For example: 

• statistics module has to know when the particular relay goes on and off,
• pressing a button has to change display content, and it also has to suspend services that would consume many CPU cycles, for example, temperature reading from the sensor,
• after some time temperature reading has to be renewed, 
• and so on...

Every module is connected to Message Bus and can register for particular events and can produce any events.

For example, pressing Next button results in the following flow:

Some components have some tasks that need to be executed periodically. We could call their corresponding methods from the main loop since we have Message Bus it's only necessary to propagate the right event:

LIBS

The following libs are required to compile Thermostat:

• https://github.com/maciejmiklas/Thermostat
• https://github.com/milesburton/Arduino-Temperature-Control-Library
• https://github.com/maciejmiklas/ArdLog.git

Weather Station based on Arduino and NodeMCU

We are about to build a weather station with a forecast for three days, including a clock with local time and date.

The whole project is based on Arduino, and NodeMCU provides access to the Internet over WiFi. The display is built from single LEDs. Since they can be really bright, it adopts illumination based on light conditions.

Let's start from the beginning ;)

I've found those 8x8 LED modules:

So I've decided to combine a few to build a display. In my case, there are 3 lines, each consisting of 8 modules, 24 in total, this gives us 1532 single LEDs! 

To drive a single module, I've chosen MAX72xx, I also wanted to improve my soldering skills, so I've decided to go for 24 PIN DIP chips and solder them to prototype boards:

Well, that worked out pretty well when it comes to those skills, but I would recommend using LED modules combined with MAX Chip, this will save you at least a few hours, not to mention time spent afterward when a single cable gets loose ;) Such a combo module has only 3 wires instead of 16.

So we have a hardware part for our display: 24 led modules with drivers. Now it's time to light them up! I've decided to go for Arduino Mega because it has two serial ports, so it's easier to debug things (one port will be used for communication with EPS8266). You could also use Uno, in this case, you would have to daisy chain MAX chips and change addressing in the software. I've used separate lines for each Max chip, but Uno just does not have enough digital output pins.

I was looking for API that will join all led modules into one canvas, so you can print sprites without bothering with transitions between LED modules. I did not find anything that would make me happy, so I implemented one myself. It provides not only a simple canvas but fonts and a few animations. Basically, everything that will be needed to display time and weather.

So ... we have a display and API to control it. Now we need to get the date and weather. Arduino does not support Internet connectivity and does not have enough resources to process incoming data. So I've decided to use NodeMCU. With a few Lua scripts, I could implement a simple API accessible over the serial port. Arduino connects over it with NodeMCU, obtains time, date, and weather, and displays it.

To provide the date, NodeMCU connects with the NTP server and receives UTC time, afterwards it calculates the local date from it. I could use one of the Internet services to grab the date, but I was looking for a solution that would remain stable for a long time. Those services can change their API or just go offline, but the NTP protocol will remain unchanged. In the worst case, you will have to change the server name. The current implementation also supports failover through many different servers, so you will probably not have to bother with it soon ;)

Getting the weather was a bit tricky because I had to find an API that would return a response small enough so it could be parsed by NodeMCU. I've decided to use Yahoo Weather. They provide a nice REST API with a small and simple response. Hopefully, they will keep interfaces stable for a long time.....

Putting things together

Hardware

First, you have to build the display, I've already described it in this post: Arduino LED Display . You can use the same pin numbers on Mega to not have to alter the software.

After the display is ready, you should connect ESP8266 and the photoresistor. The Schematic below contains all hardware elements together:

The area on the top right corner is the display itself - 8x3 LED modules. Every single module exposes 3 PINs, those are MAX 72xxx PINs responsible for communication over SPI.
Here you will find an exact schematic of the single module, including SPI PINs:

The next one is just another representation:

Those last two schematics can be found here: https://github.com/maciejmiklas/LEDDisplay/tree/master/doc/fritzing

Now it's time to build software.

Software for Arduino Mega

You need to compile this project https://github.com/maciejmiklas/LEDClock and upload into Arduino.
In order to compile it, you will need to include SPI module and two other libraries: https://github.com/maciejmiklas/LEDDisplay and https://github.com/maciejmiklas/LEDDisplay

Here is a compiled software for those with the same hardware setup.

You can use Sloeber for compilation, just follow the steps:

1. Install Sloeber from http://eclipse.baeyens.it
2. Create a new Arduino sketch and name it: LEDClock 
   
3. Confirm the next screens until it asks you about code, select cpp: 
   
4. Finish it, and you should get something like that:
   
5. Clone the LEDClock project from https://github.com/maciejmiklas/LEDDisplay and move its content into the Sloeber project folder. This operation should replace two files: LEDClock.h and LEDClock.cpp. Now we have the Sloeber cpp project with the right main files. Those wired steps were necessary because I did not want to check into GitHub IDE-specific files. This is our structure now:
   
6. There are still compilation errors, we will now add missing libraries
7. Clone LED Display API: https://github.com/maciejmiklas/LEDDisplay and import into the project:
   
8. Repeat this procedure for Logger: https://github.com/maciejmiklas/ArdLog
9. Imported project LEDDisplay has a subfolder: examples. We must exclude it from compilation because it contains files with the main-loop, which will disturb Sloeber. Select Properties on folder examples and check: Exclude resource from the build:
   
10. Now you should have the following structure:
    
11. You can upload it!

Software for NodeMCU/ESP8266

I am using EPS8266 with a Lua interpreter, this combination is called NodeMCU.

To provide weather and time to Arduino, you must clone NodeMCUUtil, modify a few scripts and finally upload those into NodeMCU. Below you will find exact instructions: 

1. Compile firmware for NodeMCU so that it has all required modules. Here you will find instructions on it, which are the required modules: file, gpio, net, node, tmr, uart, wifi and cjson.
2. Clone project containing Lua scripts: https://github.com/maciejmiklas/NodeMCUUtils
3. Edit serialAPIClock.lua and set UTC offset for your location. This will be required to calculate local date from UTC time. For most European countries it's already set to correct value. For US you will have to replace require "dateformatEurope" with require "dateformatAmerica" and rename all method calls from setEuropeTime to setAmericaTime
4. Edit yahooWeather.lua and provide city and country that you would like to have weather for.
5. Create a new file called: credentials.lua and specify login data for WiFi connection, it's just one line, for example cred = {ssid = 'openwifi', password = '123456789'}
6. Upload all Lua scirpts from main project's folder into NodeMCU:
• credentials.lua
• dateformat.lua
• dateformatAmerica.lua
• dateformatEurope.lua
• ntp.lua
• ntpClock.lua
• serialAPI.lua
• serialAPIClock.lua
• serialAPIYahooWeather.lua
• wlan.lua
• yahooWeather.lua
7. Now for the final touch, we need the init-file that will be executed right after NodeMCU boots up. In our case, we use only the Serial Port to expose weather and clock API. This also means that once our API is registered, it's impossible to execute standard NodeMCU commands, like file upload. For this reason init-script has two seconds delay, during this time you can still upload files, or just remove current init.lua file. Init-files are there: NodeMCUUtils/init/serialInit
• init.lua
• serialInit.lua

Github repos used in this project

• LED Clock: https://github.com/maciejmiklas/LEDClock/releases/tag/v1.0.0
• LED Display: https://github.com/maciejmiklas/LEDDisplay/releases/tag/v1.1.0
• Arduino Logger: https://github.com/maciejmiklas/ArdLog/releases/tag/v1.1.0
• NodeMCU weather and clock: https://github.com/maciejmiklas/NodeMCUUtils/releases/tag/v1.4.0

NodeMCU (ESP8266) as provider of Yahoo Weather

I've developed a small library that can be used to access Yahoo Weather over NodeMCU and then provide it over the serial port. This can be connected to anything with a serial port - for example, Arduino. In such case, you have an easy way of accessing the weather for your Arduino projects :)

When connecting ESP8266 to Arduino, you need a voltage divider to convert 5v from Arduino to 3.3v.

Here is the source code: https://github.com/maciejmiklas/NodeMCUUtils 

yahooWeather.lua

yahooWeather.lua script provides access to Yahoo Weather.
yaw.start() will obtain weather immediately and refresh it every yaw.syncPeriodSec seconds. Weather data itself is stored in yahooWeather.lua -> yaw.weather, you will find their further documentation. 

In the example below, you can see how to get wether for Munich and update it periodically:

require "yahooWeather"
require "wlan"

yaw.city = "munich"
yaw.country = "de"   
wlan.setup("free wlan", "12345678")

-- update weather every 17 minutes
yaw.syncPeriodSec = 1020

yaw.responseCallback = function()
    print("Weather for today:", yaw.weather[1].date) 
    print(yaw.weather[1].low, yaw.weather[1].high, yaw.weather[1].text)

    print("Weather for tomorrow:", yaw.weather[2].date) 
    print(yaw.weather[2].low, yaw.weather[2].high, yaw.weather[2].text)
end

-- start weather update timer
yaw.start()

Weather for today:  01 Sep 2016
18  25  Partly Cloudy

   
Weather for tomorrow:   02 Sep 2016

16  25  Partly Cloudy

Serial API

The script below registers Serial API Interface providing access to weather. The weather itself will get updated every 17 minutes.

require "serialAPI"
require "serialAPIYahooWeather"

yaw.city = "munich"
yaw.country = "de"
yaw.syncPeriodSec = 1020 -- 17 min
-- setup wlan required by NTP clokc
wlan.setup("fred", "1234567890")

-- start serial API by enabling gpio and uart
sapi.start()

-- start yahoo weather with serial API
yaw.start()

Now you can execute a few simple commands over serial:

# weather description for today
>YF1 text
Rain And Snow

# weather description for tomorrow
>YF2 text
Showers

# not existing command
>YF1 min
ERR:serialAPIYahooWeather.lua:6: attempt to concatenate field '?' (a nil value)

# max temp for tomorrow 
>YF2 low
1

# weather date for today
>YF1 date
09 Nov 2016

You can find the whole API description in serialAPIYahooWeather.lua

NodeMCU(ESP8266) as provider of NTP date and time over serial port

I've been looking for a date and time source for my Arduino projects. Arduino has limited resources, and I would like to use them for the actual work. So I've decided to use NodeMCU with Lua to access date and time from the internet, format it in a way that meets my requirements, and provide such results over the serial port.

There are many ways to access date and time from the internet, but mainly those interfaces change over time, and code using them must be adopted. I was looking for something that would last for years without interaction from my side. That's why I've decided to use Network Time Protocol. There is only one catch: it's UTC time without a date. It was not such a big deal because there are a lot of algorithms out there, so I've just adopted one of them.

Now it's possible to obtain local date and time from NodeMCU over the serial port, and the whole thing is reliable and will last for many years without touching it :)

Here is the source code: https://github.com/maciejmiklas/NodeMCUUtils

The implementation is divided into small building blocks; putting them together will give you the desired serial API. This is just a short overview:

• Date Format - calculates local date and time based on the timestamp
• WiFi Access  - simple facade for connecting to WiFi
• NTP Time - obtains UTC timestamp form given NTP server
• NTP Clock - keeps actual UTC timestamp and synchronizes it periodically with the NTP server
• Serial API - finally, this one provides API for date and time

Date Format

Provides functionality to get local date and time from timestamp given in seconds since 1970.01.01

For such code:

collectgarbage() print("heap before", node.heap())

require "dateformat"
require "dateformatEurope"
local ts = 1463145687
df.setEuropeTime(ts, 3600) -- function requires GMT offset for your city

print(string.format("%04u-%02u-%02u %02u:%02u:%02d", 
    df.year, df.month, df.day, df.hour, df.min, df.sec))
print("DayOfWeek: ", df.dayOfWeek)

collectgarbage() print("heap after", node.heap())

You will get this output:

heap before 44704
2016-05-13 15:21:27
DayOfWeek:  6
heap after  39280

To format the date for the USA, you have to replace "dateformatEurope" with "dateformatAmerica" and call setAmericaTime instead of setEuropeTime. Functionality is divided into small scripts to save some RAM.

WiFi Access

It's a simple facade for connecting to WiFi. You have to provide connection credentials and a function that will be executed after the connection has been established.

execute(...) connects to WiFi, which can take some time. You can still call this method multiple times. In such cases, callbacks will be stored in the queue and executed after establishing a WiFi connection.

require "wlan"

wlan.debug = true

local function printAbc() 
    print("ABC")
end

wlan.setup("free wlan", "12345678")
wlan.execute(printAbc)

Configuring WiFi on:   free wlan
status  1
status  1
status  5
Got WiFi connection:   172.20.10.6 255.255.255.240 172.20.10.1
ABC

NTP Time

This simple facade connects to a given NTP server, requests UTC time from it, and once a response has been received, it calls the given callback function. 

The example below executes the following chain: WiFi -> NTP -> Date Format. 

So in the first step, we create a WLAN connection and register a callback function that will be executed after the connection has been established. This callback function requests time from the NTP server (ntp.requestTime). 

On the ntp object, we are registering another function that will get called after the NTP response has been received: printTime(ts).

collectgarbage() print("RAM init", node.heap())

require "wlan"
require "ntp"
require "dateformatEurope";

collectgarbage() print("RAM after require", node.heap())

ntp = NtpFactory:fromDefaultServer():withDebug()
wlan.debug = true

local function printTime(ts) 
    collectgarbage() print("RAM before printTime", node.heap())
    
    df.setEuropeTime(ts, 3600)
    
    print("NTP Local Time:", string.format("%04u-%02u-%02u %02u:%02u:%02d", 
        df.year, df.month, df.day, df.hour, df.min, df.sec))
    print("Summer Time:", df.summerTime)
    print("Day of Week:", df.dayOfWeek)
    
    collectgarbage() print("RAM after printTime", node.heap())
end

ntp:registerResponseCallback(printTime)

wlan.setup("free wlan", "12345678")
wlan.execute(function() ntp:requestTime() end)

collectgarbage() print("RAM callbacks", node.heap())

and console output:

RAM init    43328
RAM after require   30920
Configuring WiFi on:   free wlan
RAM callbacks   30688
status  1
status  1
status  5
Got WiFi connection:   172.20.10.6 255.255.255.240 172.20.10.1
NTP request:    pool.ntp.org
NTP request:    194.29.130.252
NTP response:   11:59:34
RAM before printTime    31120
NTP Local Time: 2016-07-12 13:59:34
Summer Time: 
Day of Week:    3
RAM after printTime 30928

NTP Clock

This script provides functionality to run a clock with the precision of one second and to synchronize this clock every few hours with the NTP server. 

In the code below, we first configure WiFi access. Once the WiFi access has been established, it will call ntpc.start(). This function will start a clock synchronizing with a given NTP server every minute. Now you can access actual UTC time in seconds over ntpc.current. To show that it's working, we have registered a timer that will call: printTime() every second. This function reads current time as ntpc.current and prints it as local time. 

collectgarbage() print("RAM init", node.heap())

require "dateformatEurope";
require "ntpClock";
require "wlan";

collectgarbage() print("RAM after require", node.heap())

ntpc.debug = true
wlan.debug = true

wlan.setup("free wlan", "12345678")
wlan.execute(function() ntpc.start("pool.ntp.org", 60) end)

local function printTime() 
    collectgarbage() print("RAM in printTime", node.heap())
    
    df.setEuropeTime(ntpc.current, 3600)
    
    print("Time:", string.format("%04u-%02u-%02u %02u:%02u:%02d", 
        df.year, df.month, df.day, df.hour, df.min, df.sec))
    print("Summer Time:", df.summerTime)
    print("Day of Week:", df.dayOfWeek)
end

tmr.alarm(2, 30000, tmr.ALARM_AUTO, printTime)

so this is the output:

RAM init    43784
RAM after require   29408
Configuring WiFi on:    free wlan
status  1
status  5
Got WiFi connection:    192.168.2.113   255.255.255.0   192.168.2.1

NTP request:    pool.ntp.org
NTP request:    195.50.171.101
NTP response:   17:09:46

RAM in printTime    29664
Time:   2016-08-08 19:10:08
Summer Time:    true
Day of Week:    2

RAM in printTime    29808
Time:   2016-08-08 19:10:38
Summer Time:    true
Day of Week:    2

NTP request:    pool.ntp.org
NTP request:    195.50.171.101
NTP response:   17:10:46

RAM in printTime    29680
Time:   2016-08-08 19:11:08
Summer Time:    true
Day of Week:    2

RAM in printTime    29808
Time:   2016-08-08 19:11:38
Summer Time:    true
Day of Week:    2

NTP request:    pool.ntp.org
NTP request:    131.188.3.221
NTP response:   17:11:46

RAM in printTime    29680
Time:   2016-08-08 19:12:08
Summer Time:    true
Day of Week:    2

RAM in printTime    29808
Time:   2016-08-08 19:12:38
Summer Time:    true
Day of Week:    2

Serial API

Serial API exposes a simple interface that provides access to diagnostic info and date to be accessed outside NodeMCU - for example, by Arduino.

Serial API is divided into a few Lua scripts. Loading of each script will automatically add new API commands:

- serialAPI.lua - has to be always loaded. It initializes the serial interface with a few diagnostics commands.

- serialAPIClock.lua - access to clock including date formatter.

Each script above registers a set of commands as keys of scmd table and contains further documentation.

The example below provides date access over the serial port:

require "credentials"
require "serialAPI"
require "serialAPIClock"

ntpc.syncPeriodSec = 900 -- 15 min
sapi.baud = 115200

-- setup wlan required by NTP clock
wlan.setup("free wlan", "12345678")

-- start serial API by enabling gpio and uart
sapi.start()

-- start NTP synchronization
ntpc.start("pool.ntp.org")

Here are few Serial API commands and their responses:

# free ram
>GFR
10664

# WiFi status
>GWS
5

# date and time (24h) in format: yyyy-mm-dd HHLmm:ss
>CF1
2016-09-16 10:45:25

# date in format: yyyy-mm-dd
>CH2
10:45:59

Firmware

Executing multiple scripts can lead to out-of-memory issues on NodeMCU. One possibility to solve it is to build custom firmware containing only a minimal set of node-mcu modules: cjson, file, gpio, net, node, tmr, uart, and WiFi. This blog provides a detailed upgrade procedure: http://maciej-miklas.blogspot.de/2016/08/installing-nodemcu-v15-on-eps8266-esp.html

Upgrading NodeMCU Firmware to v1.5 on EPS8266 (ESP-12E)

I've been updating firmware on my NodeMCU a few times without issues - at least up to 0.9
The installation procedure has changed with SDK 1.5 - I've spent a few hours before I got it right, so maybe this post will help you save time.

Here you will find all the required information, but depending on your chip manufacturer, you will need different combinations of some options, and this is exactly the part when it's getting tricky.

Since things are changing rapidly, I will point out exact versions in this tutorial so that it's clear what was working with what at the time of this writing.

Installing esptool

Download esptool v1.1 https://github.com/themadinventor/esptool/releases/tag/v1.1 and unzip it.
This will give you the possibility to execute esptool.py from the command line - we will use it below.

Determining the version of your ESP8266 chip

As usual, there are different versions, and probably you will have to modify flashing commands if you have a different one.

sudo python esptool.py -p /dev/tty.wchusbserial1410 flash_id

esptool.py v1.2-dev
Connecting...
Manufacturer: e0
Device: 4016

Well, it's obviously ESP-12E with 4MB flash:

How do you tell? I've no idea - just Google "Manufacturer: e0 Device: 4016" and check what people think.

Building custom firmware 

Now we have to obtain firmware containing the Lua interpreter for our chip - it's everything you will need to upload Lua scripts over serial. 

Basically, you will need two binary files: 

• custom firmware - something like: nodemcu-dev-7-modules-2016-08-10-10-43-59-integer.bin
• initial-data-block (esp_init_data_default.bin)

Obviously, you will need matching versions. 

There are two options to get those files: 

• you can download it from my github repo - this will give you SDK 1.5.4.1
• you can also use cloud service to build latest release. The link to latest initial-data-block can be found here - just search for esp_init_data_default.bin

I would prefer the latest version - as usual. But in case of some problems, you can try using the version provided on my github repo. At least this one was working in my case with ESP-12E, this might help you to locate a problem.

Erase flash

Put your chip into flash mode as described here. If you have a development board, it should have two buttons: flash and reset. In such case, press flash, and without releasing it, press reset, after that, release flash.

Now you must determine and change the com port in all commands in this tutorial. In my case, it's /dev/tty.wchusbserial1410

Let's finally erase flash!

sudo python esptool.py -p /dev/tty.wchusbserial1410 erase_flash
esptool.py v1.1
Connecting...
Erasing flash (this may take a while)...

Now disconnect ESP from usb port, so that it can fully reset - pressing reset button does not have the same effect!

Flashing new firmware

Put your chip again into flash mode.
I am assuming that you have in your current directory two binary files: one with firmware (nodemcu-dev-7-modules-2016-08-10-10-43-59-integer.bin) and a second with init-stuff (esp_init_data_default.bin).

sudo python esptool.py -p /dev/tty.wchusbserial1410 write_flash -fm dio -fs 32m 0x000000 nodemcu-dev-7-modules-2016-08-10-10-43-59-integer.bin 0x3fc000  esp_init_data_default.bin

esptool.py v1.1
Connecting...
Running Cesanta flasher stub...
Flash params set to 0x0240
Writing 385024 @ 0x0... 385024 (100 %)
Wrote 385024 bytes at 0x0 in 33.4 seconds (92.2 kbit/s)...
Writing 4096 @ 0x3fc000... 4096 (100 %)
Wrote 4096 bytes at 0x3fc000 in 0.4 seconds (80.0 kbit/s)...
Leaving...

Testing....

I've used ESPlorer, and it looks just fine.