My Soap Service Proxy Codeunit

Up to now we in Advania have been using the method described here on my blog to connect to most of the Soap web services that we needed to integrate with.

The problem with this method is that we have to manage a lot of DLLs.  This has caused some issues and problems.

Another thing is that we are moving to AL.  And in AL we can’t just throw in a custom DLL to do all the work.

In C/AL We can do this with standard dotnet objects

AL code to do the same with the built in AL objects but that code is not much shorter.

With a custom proxy DLL the code would be

With this example we can easily see why we have chosen to create a proxy DLL for most of the Soap services.

I wanted to find a way to make things easier in AL and I remembered having dealt with C/AL objects by Vjeko from some time ago.  I took another look and that code helped me to get started.

The result is a Soap Proxy Client Mgt. Codeunit in C/AL that I have sent to Microsoft’s cal-open-library project asking to have this code put into the standard C/AL library.

Using this Codeunit the code will be like this.

What about AL?

For now this C/AL Codeunit is not in the standard CRONUS database.  I need to import the C/AL code and make sure that AL will be able to use that Codeunit.  You can see how to do this in my last blog post.

This C/AL Code will directly convert to AL and is ready to use.

More examples on how to use this Proxy Codeunit will follow.  Stay tuned…

C/AL and AL Side-by-Side Development with AdvaniaGIT

Microsoft supports Side-by-Side development for C/AL and AL.  To start using the Side-by-Side development make sure you have the latest version of AdvaniaGIT add-in for Visual Studio Code and update the PowerShell scripts by using the “Advania: Go!” command.

When the Business Central environment is built use the “Advania: Build C/AL Symbol References for AL” to enable the Side-by-Side development for this environment.  This function will reconfigure the service and execute the Generate Symbol References command for the environment.  From here on everything you change in C/AL on this environment will update the AL Symbol References.

So let’s try this out.

I converted my C/AL project to AL project with the steps described in my previous post.  Then selected to open Visual Studio Code in AL folder.

In my new Visual Studio Code window I selected to build an environment – the Docker Container.

When AdvaniaGIT builds a container it will install the AL Extension for Visual Studio Code from that Container.  We need to read the output of the environment build.  In this example I am asked to restart Visual Studio Code before reinstalling AL Language.  Note that if you are not asked to restart Visual Studio Code you don’t need to do that.

After restart I can see that the AL Language extension for Visual Studio Code is missing.

To fix this I execute the “Advania: Build NAV Environment” command again.  This time, since the Container is already running only the NAV license and the AL Extension will be updated.

Restart Visual Studio Code again and we are ready to go.

If we build new environment for our AL project we must update the environment settings in .vscode\launch.json.  This we can do with a built in AdvaniaGIT command.

We can verify the environment by executing “Advania: Check NAV Environment”.  Everything should be up and running at this time.

Since we will be using Side-by-Side development for C/AL and AL in this environment we need to enable that by executing “Advania: Build C/AL Symbol References for AL”.

This will take a few minutes to execute.

Don’t worry about the warning.  AdvaniaGIT takes care of restarting the service.  Let’s download AL Symbols and see what happens.

We can see that AL now recognizes the standard symbols but my custom one; “IS Soap Proxy Client Mgt.” is not recognized.  I will tell you more about this Codeunit in my next blog post.

I start FinSql to import the Codeunit “IS Soap Proxy Client Mgt.”

Import the FOB file

Close FinSql and execute the “AL: Download Symbols” again.  We can now see that AL recognizes my C/AL Codeunit.

Now I am good to go.

REST Web Services using Json and requiring authentication

But first…

Registration for NAV TechDays 2017 have been opened.  I will do a workshop on web services and json.  I will be using both C/AL and AL with VS Code in this workshop.

Make sure to register for the conference and if possible go to one or two of the workshops.

Now to the topic.  Yesterday I started to develop an integration solution for bokun.io.  Their API is RESTful and uses Json file formats.  It also requires authentication.

In a project like this I usually start by using the OCR Service Setup from standard NAV.  Create a Setup table and a page.

Looking at the API documentation we can see that we need to use HmacSHA1 with both Access Key and Secret Key to authenticate.  In other project I used HmacSHA256 with the Access Key for the Azure API.

First part of the authentication is the time stamp created in UTC.  I find it easy to use the DateTime DotNet variable to solve this.  There are two different formatting I needed to use.

REST service normally just use GET or POST http methods.  The authentication is usually in the request headers.  This is an example from bokun.is

The GetSignature function is

The Secret Key string and the Signature is converted to a byte array.  The Crypto class is constructed with the Secret Key Byte Array and used to compute hash for the Signature Byte Array. That hash is also a byte array that must be converted to a base64 string.  This will give you the HmacSHA1 signature to use in the request header.

My Azure project is using HmacSHA256 but the code is similar.

Azure displays the Access Keys in base64 format while bokun.is has a normal string.

A little further down the line I choose not to use XML Ports, like I did here, but still convert Json to Xml or Xml to Json.

I use the functions from Codeunit “XML DOM Management” to handle the Xml.  This code should give you the general idea.

 

 

NAV Http Web Request

In my post about Json and Rest web services I showed how to use the standard Codeunit no. 1297 for web service communication.

Today I was asked to do this in NAV 2015.  I must admit, I forgot that this Codeunit was not available in NAV 2015.

So I made one.

This one has identical functionality to the one delivered with NAV 2016.  To catch and handle the errors I use the NAV Web Request Add-in that I created and published here on my blog.

Now I can easily move that Json code down to NAV 2015.

Download here –> COD1297-NAV2015

My first Dynamics 365 Extension – step by step – sixth step

Assisted Setup and Permissions.

But first lets look at the Extension Management Codeunit.

I want to store the appId in my Extension.  This appId is used in several tables in my tenant.  This appId must always be the same for this solution even if the solution is updated.  In my last post you can see that I am also setting my appId parameter in the Git repository.  There are few easy ways to get your appId.  NAV has the function CREATEGUID.  Powershell has the function New-Guid.  But perhaps the easiest way is to use this online Guid generator.

In OnNavAppUpgradePerDatabase – executed once when the Extension is installed, I want to assign my Extension Setup permission set to the user installing.  This is required to be able to run the Assisted Setup after installation.

In OnNavAppUpgradePerCompany – executed once in every company when the Extension is installed, I want to restore data for some of the Extension tables and delete rest of the archive data.  Even if I am just using tables as a temporary tables I still need to define what to do with the non-existing-archived data.  In here I also want to remove the persistent Assisted Setup record in every company.

So, why would I like to remove the Assisted Setup record?  Not because I like to make the user execute the setup every time the Extension is updated.  The Extension has its own Setup table.  In that Setup table I store the setup information belonging to this Extension and the Setup data is restored from archive during the installation as you can see above.

In the current release the Assisted Setup record is a database record.  I know that Microsoft intends to change this and use the same discovery pattern that is used in Service Connections.  When that happens the Assisted Setup record will be temporary only.  So, by designing my Extension in this way I make sure that I will still support Assisted Setup after this change.

In NAV 2016 we had the Mini Role Center, page 9022.  Today this Role Center has been updated to fit Dynamics 365 for Financials.

From here the user can access the Extension installation we covered in the last post, and we can also access the Assisted Setup & Tasks.

I add my Assisted Setup by subscribing to the OnOpenPage Event in the above page.

Looking at the Assisted Setup record we can see that is has a media field for an Icon.  NAV setup data includes icon for the standard Assisted Setup items – so must we.

I decided to ship my icons with the code.  Each icon has a dedicated Codeunit.

To fit an icon into a Codeunit we need to convert the binary data to base64 data.  This we can do with the following code.

This allows me to create a Codeunit like this.

And then code like this to import the icon into my help resources.

Now, back to the Assisted Setup.  I start the setup for my Extension.

This sums up what needs to be done.  The basic setup, the one done with Set Defaults, is to assign permissions to all users based on their current permissions to the tables we build our extension around.

The G/L Source Name lookup table is read from a FlowField in G/L Entries.  We can therefore expect that everyone that has read access to G/L Entries must also have read access to the lookup table. The data in the lookup table is updated when ever any of the four master tables is updated.  Hence, everyone that has access to modify any of the master tables must have access to modify the lookup table.

The Set Defaults function assigns the Extension permission sets to users and user groups based on the current permissions and that should be enough in most cases.  If the user likes the advanced way that possibility is available.

When the user presses Finish the wizard data and the assigned permission sets are applied to the tenant database.

When I created the Setup Wizard page I started with a copy of the Email Setup Wizard, page 1805.

The model I have created to assign permission sets to users and user groups with a Wizard can easily be modified to fit any Extension.  Remember that all this code will soon be available on my GitHub account.

 

Using REST/Json web services from NAV

One of my most popular blog entry is the one about Json.  I have also had some questions outside this website about this topic.

This week I got a task.  We need to communicate with a payment service that uses REST web services and Json file format.

posapi

I got a document describing the service.  Some methods use GET and some use POST.  Here is how I did this.

In the heart of it all I use Codeunit 1297, “Http Web Request Mgt.”.

getaccesstoken

Every time we talk to this POS API we send an Access Token.  If we don’t have the token in memory (single instance Codeunit), we need to get a new one.  That is what the above code does.

The ParameterMgt Codeunit is what I want to focus on.  You can see that I start by inserting my “Authorization Key” into the RequestBodyBlob.  As usual, I use the TempBlob.Blob to get and set my unstructured data.

setapirequest

The interesting part here is that I use an XMLPort to create the data I need to post to the Api.

apiauthenticatexml

A simple one in this example, but nothing says it can’t be complex.  Then I convert the Xml to Json with a single function.

converttojson

The last TRUE variable means the the Document Element will be skipped and the Json will look like it is supposed to.

apikey

The REST service response is Json.

token

And to read the Json response we take a look at the GetAccessToken function.

getaccesstokenfunction

Here I start by converting from Json to Xml.

convertfromjson

And make sure my Document Element name is “posApi”.

apiaccesstokenxml

And I have the result.

As you can see from the documentation some of the Json data is more complex.  This method will work nevertheless.

For more complex date I always create tables that matches the Json structure.  These table I use temporary through the whole process so the don’t need to be licensed tables.  Here is an example where this XMLPORT

getauthorization

will read this Json

getauthorizationjson

I suggest that with our current NAV this is the easiest way to handle REST web services and Json.

 

Asynchronous web services in NAV

Asynchronous methods are very useful for web services.  To be able to start a process with one method and then check the status with another opens a lot of possibilities.

There are two known patterns in NAV that support asynchronous methods;  one is to use the STARTSESSION function to execute the business logic in another thread, the other is to create a job queue entry and let NAS handle the task.

I was talking to a client this morning and suggested using the asynchronous web services for his tasks.  I decided to write some code and test the functionality to see if my theory was working.

The first thing I need is a table for the requests.  This table contains the identification for the queue, the status and other details.

ProcessQueueTable

A single web service method is used to create a new queue entry.

AsyncWebService

I like to use the new TryFunction for my web services.  Note that when the queue has been inserted I fire an event with the newly created queue entry.  This means that I can extend this web service with events without having to modify the web service signature.

Each process needs a dedicated Codeunit.  That Codeunit is built on top of the queue record and thereby using the parameter table pattern.

TheAsyncJob2

Note that this Codeunit is also using the TryFunction and taking care of the rollback if that function fails.

The process Codeunit catches the integration event and checks the process code before starting then selected task.

ProcessStarter2

So, if I ask the web service to start a job called StartAsyncAdjustCostItemEntriesProcess, NAV will start a new session to execute the adjust cost for item entries.  The process that creates the queue and starts a new session is very quick so the response from the web service is almost instant.

Then we wait for a moment and ask the web service about the status for the newly created queue.

CheckQueueStatus

Again using the TryFunction to make sure that my web service will have a proper response to the query.

It is quite easy to extend this module by catching the OnNewQueueInserted event for each process code you build support for.

A sample C# code that I used to test this looks like this

CSharpDemoCode2

As you can imagine we can put anything into the ProcessData variable.  By using Base64 encoding anything can be converted to a text variable.

I hope this will turn on some lights and you will be able to use this in your daily work.

Attached is a zip file with the NAV objects and the C# project.

NAVAsyncWebService

 

 

 

 

JSON meets NAV

I have been using SOAP services over the last years.  Only recently the RESTful web services have become more and more popular in my integration work.  Wikipedia says:

In computing, Representational State Transfer (REST) is a software architecture style for building scalable web services. REST gives a coordinated set of constraints to the design of components in a distributed hypermedia system that can lead to a higher performing and more maintainable architecture.

RESTful systems typically, but not always, communicate over the Hypertext Transfer Protocol with the same HTTP verbs (GET, POST, PUT, DELETE, etc.) which web browsers use to retrieve web pages and to send data to remote servers. REST interfaces usually involve collections of resources with identifiers, for example /people/paul, which can be operated upon using standard verbs, such as DELETE /people/paul.

As we are used to XML as the body for our SOAP messages we can also use XML as the body for a RESTful web service.  I just finished writing a code to communicate with Azure from NAV.  This communication was using RESTful web services and XML.

So, what is JSON?  Wikipedia says:

JSON, (canonically pronounced /ˈdʒeɪsən/ JAY-sən; sometimes JavaScript Object Notation), is an open standard format that uses human-readable text to transmit data objects consisting of attribute–value pairs. It is the primary data format used for asynchronous browser/server communication (AJAJ), largely replacing XML (used by AJAX).

Although originally derived from the JavaScript scripting language, JSON is a language-independent data format. Code for parsing and generating JSON data is readily available in many programming languages.

The JSON format was originally specified by Douglas Crockford. It is currently described by two competing standards, RFC 7159 and ECMA-404. The ECMA standard is minimal, describing only the allowed grammar syntax, whereas the RFC also provides some semantic and security considerations. The official Internet media type for JSON is application/json. The JSON filename extension is .json.

With JSON it is possible to deliver similar data structure as with XML.  JSON on the other hand requires a much less metadata.  Here is an example JSON from Wikipedia:

[code lang=”javascript”]{
"firstName": "John",
"lastName": "Smith",
"isAlive": true,
"age": 25,
"address": {
"streetAddress": "21 2nd Street",
"city": "New York",
"state": "NY",
"postalCode": "10021-3100"
},
"phoneNumbers": [
{
"type": "home",
"number": "212 555-1234"
},
{
"type": "office",
"number": "646 555-4567"
}
],
"children": [],
"spouse": null
}[/code]

There is not a good support for JSON in native .NET from Microsoft.  However, with Visual Studio, Microsoft installs an external DLL in to the folder “C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\PrivateAssemblies”

Newtonsoft

With this Json.NET in Dynamics NAV Add-ins folder we now have some way to handle JSON files.  Using this Add-in I created a NAV Codeunit to manage JSON text.

This Codeunit contains functions to build a JSON document, like

[code lang=”csharp”]
StartJSon;
AddToJSon(‘newssn’,CompanyInformation."Registration No.");
AddToJSon(‘billtossn’,BillToCustNo);
AddToJSon(‘newcompanyname’,CompanyInformation.Name);
AddToJSon(‘newemail’,CompanyInformation."E-Mail");
AddToJSon(‘register_einvoice’,EInvoiceEnabled);
AddToJSon(‘register_supdoc’,SupDocEnabled);
AddToJSon(‘register_natreg’,NRLookupEnabled);
EndJSon;
Json := Json.Copy(GetJSon);[/code]

A function to import values from a JSON document to a temporary table, like

[code lang=”csharp”]
ReadJSon(String,TempPostingExchField);

WITH TempPostingExchField DO BEGIN
SETCURRENTKEY("Line No.","Column No.");
IF FIND(‘-‘) THEN REPEAT
SETRANGE("Column No.","Column No.");
InsertFileDetails(TempPostingExchField,WebServiceURL);
FINDLAST;
SETRANGE("Column No.");
UNTIL NEXT = 0;
END;[/code]

Or just a simple way to return a single value from a simple JSON string, like

[code lang=”csharp”]FileName := GetValueFromJsonString(String,’filename’);[/code]

With these functions NAV should be able to handle JSON files without any problems.

Now you can add JSON handling to your arsenal.

Json Codeunit and required add-ins

Using NetTcpPortSharing for NAV Servers

I just came back from three weeks vacation yesterday.  During my vacation I had made a decision to implement Tcp Port Sharing for the Instance Administration tool used in Advania Azure.

Early last year I published a function that uses the sc.exe to modify a NAV Service startup type.  When a NAV Service is installed and configured in setup, the startup type is Automatic (Delayed Start).  However, create a new service with Powershell New-NavServerIntance and the statup type is Automatic without the (Delayed Start).

To enable Tcp Port Sharing that same sc.exe function is needed.  Interestingly, after I had finished the task and was reading NAV blogs I saw that Waldo just published a powershell function on his blog to do just this.

The script lines I used and added to my Instance Administration powershell scripts are based on my fist sc.exe function but not using the function it self.  Now when a new NAV service is created by the tool the startup type is modified and if so selected by the deployment settings, the Tcp Port Sharing is also activated.

By default, the Tcp Port Sharing service is disabled.
servicedisabled

The startup type should be changed to Manual.  This can be done manually or by an administrative powershell script.

[code lang=”powershell”]#Set Startup Mode for NetTcpPortSharing to Manual
$command = ‘sc.exe \\$Computer config "NetTcpPortSharing" start= demand’
$Output = Invoke-Expression -Command $Command -ErrorAction Stop
if($LASTEXITCODE -ne 0){
Write-Error "$Computer : Failed to set NetTcpPortSharing to manual start. More details: $Output"
}
[/code]

Similar script is used to update the existing NAV Services to both delayed start and Tcp Port Sharing dependency.

[code lang=”powershell”]

#Stop NAV Server Instances
Get-NAVServerInstance | Set-NAVServerInstance -Stop
#Update Startup Type and Dependency on NAV Server Instances
Get-NAVServerInstance | foreach {
$Service = $_.ServerInstance
Write-Host "Working on service $Service"
$Computer = ‘LOCALHOST’
$command = ‘sc.exe \\$Computer config "$Service" start= delayed-auto’
$Output = Invoke-Expression -Command $Command -ErrorAction Stop
if($LASTEXITCODE -ne 0){
Write-Error "$Computer : Failed to set $Service to delayed start. More details: $Output"
}
$command = ‘sc.exe \\$Computer config "$Service" depend= NetTcpPortSharing/HTTP’
$Output = Invoke-Expression -Command $Command -ErrorAction Stop
if($LASTEXITCODE -ne 0){
Write-Error "$Computer : Failed to set $Service TcpPortSharing. More details: $Output" -foregroundcolor red
}

}
#Start NAV Server Instances
Get-NAVServerInstance | Set-NAVServerInstance -Start
[/code]

It should be obvious that the above script can also use the Set-ServiceStartupMode from my blog and the Enable-NAVServerInstancePortSharing function on Waldo’s blog. That would be a cleaner code and more in line with what we would like to see.

Again quoting Waldo from his previous blog, “When you’re using a dedicated service account, things might become a slight more difficult”.  That is exactly my case, I am using a dedicated service account.

After enabling Tcp Port Sharing and updating the services they would not start.  Event Viewer revealed the reason.

Server instance: CRONUS
The service MicrosoftDynamicsNavServer$CRONUS failed to start. This could be caused by a configuration error. Detailed error information:System.ServiceModel.CommunicationException: The service endpoint failed to listen on the URI ‘net.tcp://mynavserver.dynamics.is:7046/CRONUS/Service’ because access was denied. Verify that the current user is granted access in the appropriate allowAccounts section of SMSvcHost.exe.config. —> System.ComponentModel.Win32Exception: Access is denied

So I started to ask Bing what I could do.  Microsoft MSDN states:

When a net.tcp binding enables port sharing (by setting portSharingEnabled =true on the transport binding element), it implicitly allows an external process (namely the SMSvcHost.exe, which hosts the Net.TCP Port Sharing Service) to manage the TCP socket on its behalf.

Hence, I need to add the Sid of my NAV Service Account to the SMSvcHost.exe.config file.  I could do this manually, but I am a programmer!

Another powershell script was born.  This one could also be converted to a function.  Before executing the script make sure to update the user and domain in the top of the script.  Be smart and execute this function before updating the NAV Services with the script above.

[code lang=”powershell”]
#Modify User and Domain to fit your environment
$UserToAdd = ‘srvNAV’
$UserDomainToAdd = ‘DYNAMICS’

#Initial Values
$UserSidFound = ‘false’
$ConfigurationSet = ‘false’

#Net.Tcp Port Sharing Service Name
$ServiceName = ‘NetTcpPortSharing’

#Get SID for the Service User
$UserSid = ([wmi] "win32_userAccount.Domain=’$UserDomainToAdd’,Name=’$UserToAdd’").SID

#Get Path for SMSvcHost.exe.config file
$SMSvcHostPath = (Get-WmiObject win32_service | ?{$_.Name -like $ServiceName} ).PathName
$SMSvcHostPathConfig = $SMSvcHostPath + ‘.config’

Write-Host "Reading XML from $SMSvcHostPathConfig"
#Read Config file
$xmlDoc = [xml] (Get-Content $SMSvcHostPathConfig)

Write-Host "Looking for access permission for $UserSid"
#Loop through allowed accounts and search for the service user Sid
$allowAccounts = Select-Xml "configuration/system.serviceModel.activation/net.tcp/allowAccounts/add" $xmlDoc
$allowAccounts | ForEach-Object {
$ConfiguredSid = $_.Node.Attributes.Item(0).Value
if ($ConfiguredSid -eq $UserSid) {$UserSidFound = ‘true’}
$ConfigurationSet = ‘true’
Write-Host "Found SID $ConfiguredSid"
}

#Act if Access Configuration is not enabled
if ($ConfigurationSet -eq ‘false’) {Write-Host "Access permission not configured"
$config = [xml] ‘<system.serviceModel.activation>
<net.tcp listenBacklog="10" maxPendingConnections="100" maxPendingAccepts="2" receiveTimeout="00:00:10" teredoEnabled="false">
<allowAccounts>
<add securityIdentifier="S-1-5-18"/>
<add securityIdentifier="S-1-5-19"/>
<add securityIdentifier="S-1-5-20"/>
<add securityIdentifier="S-1-5-32-544" />
</allowAccounts>
</net.tcp>
<net.pipe maxPendingConnections="100" maxPendingAccepts="2" receiveTimeout="00:00:10">
<allowAccounts>
<add securityIdentifier="S-1-5-18"/>
<add securityIdentifier="S-1-5-19"/>
<add securityIdentifier="S-1-5-20"/>
<add securityIdentifier="S-1-5-32-544" />
</allowAccounts>
</net.pipe>
<diagnostics performanceCountersEnabled="true" />
</system.serviceModel.activation>’

$configurationNode = $xmlDoc.DocumentElement
$newConfig = $xmlDoc.ImportNode($config.DocumentElement, $true)
$configurationNode.AppendChild($newConfig)

$allowAccounts = Select-Xml "configuration/system.serviceModel.activation/net.tcp/allowAccounts/add" $xmlDoc
$allowAccounts | ForEach-Object {
$ConfiguredSid = $_.Node.Attributes.Item(0).Value
Write-Host "Found SID $ConfiguredSid"
if ($ConfiguredSid -eq $UserSid) {$UserSidFound = ‘true’}
$ConfigurationSet = ‘true’
}

}

#Add Service User Sid if needed
if ($UserSidFound -ne ‘true’) {
$nettcp = $xmlDoc.SelectSingleNode("configuration/system.serviceModel.activation/net.tcp/allowAccounts")
$addNode = $xmlDoc.CreateElement(‘add’)
$secIden = $xmlDoc.CreateAttribute(‘securityIdentifier’)
$secIden.Value = $UserSid
$addNode.Attributes.Append($secIden)

$nettcp.AppendChild($addNode)
$xmlDoc.Save($SMSvcHostPathConfig)
Write-Host "Configuration Updated"
#Restart Service if running
if ((Get-Service NetTcpPortSharing).Status -eq "Running") {Restart-Service NetTcpPortSharing -Force}
}

[/code]

This script will search for the SMSvcHost.exe.config file, load it and check to see if the NAV Service User is already allowed access.  If not then the config file is updated and saved.  This script must be executed with administrative privileges.

Perhaps this should be what I started with, but the question; why do we need this, should be answered.

First, modifying the startup mode to delayed start is done to make sure that all the required networking and database processes have been started before the NAV Service starts.  This is very important if the SQL Server is running on the same server.  On a dedicated NAV Service server this is not as important but still recommended.

Secondly, accessing a NAV Service in most cases requires changes to a firewall.  Either to open a specific port or setting up a NAT from a public interface.  To minimize the number of ports used also minimizes the networking setup and maintenance.  If different network permissions or network access is required I recommend using separate ports for the NAV Services.