Good afternoon, Habr!


Task


My organization uses a mail server on the Kerio Connect platform, and mail servers serving their users are installed in different cities. Initially, there was no distributed structure, since the domains differ at the third level, indicating the city of the site. Everything worked and everyone was happy. One fine day - the management set a task, a common calendar of affairs between all sites!


Background


Initially, the idea was to raise the Kerio Distributed Mail Domain and he would do everything himself. It is said that a distributed domain was created, but it wasn’t there, the server was ready to synchronize calendars, folders, contacts - between domains located on the same server, but was not going to synchronize data between several servers.


Of course, I did not expect such a trick and for a long time I could not believe in the absence of the functionality I needed. Later found documentary evidence of this fact. What was very puzzled and disappointed.


The task flowed smoothly into the problem.


What were your options


  • Create two clients on different servers that exchanged the necessary data with some third-party software. It was necessary to find this very third-party software that would implement this functionality - I do not like such a rake, but it seemed that this was the only quick solution.
  • Write your own script to synchronize data between servers. The fact is that Kerio stores each object as a separate file, accordingly, it was necessary to develop a script for working with files, but in view of the sufficient number of sources, the task seemed somewhat complicated, especially since it was necessary to perform multiple checks of the correctness of the data, suddenly someone would create task in the same time interval, etc., etc.

Looking ahead, I’ll say that Kerio, although it stores the object as a separate file, is not so stupid as to ask how the file system is doing with each call to the object.


Having spent a lot of time thinking, having outlined a bunch of pieces of paper with plans “to capture the enemy’s territory,” at 6 o’clock, I made two right decisions:


  • The first decision is to do your own and not look for anything outside.
  • The second solution is to go to bed.

In the morning I woke up with one single and true thought, which was reduced to a few letters - DFS


Solution


The solution itself was as follows


  • list all servers that will participate in synchronization to Windows OS. (The part was on Linux. Mail data migration to other OS was required)
  • Determines the directory structure that will participate in synchronization - they must be identical.
  • Define all mail servers under one domain with a single DFS space.
  • Create the aforementioned distributed Kerio domain, since in my case synchronization of data is required, not only between servers but also between domains, the second can be done by Kerio server on its own. (unlike the first)
  • Set synchronized directories to DFS space.
  • To come up with some kind of crutch (after all, you cannot do without a crutch)

Implementation


Example on two mail servers (may be more)


1. Kerio Distributed domain


ITKarma picture


Master is not involved in synchronization, but this is not a prerequisite.


I won’t describe how to raise a distributed Kerio domain, there’s nothing complicated, you can study the official manul


Finally, in the administration console, you should see the following picture:


ITKarma picture


ITKarma picture


Next, I was interested in shared folders, on the Master server you can specify the following options:


ITKarma picture


ITKarma picture


Special for each domain - the server will not synchronize public folders between domains


Common for all domains - all servers will abandon the existing shared folders in each domain and create new single folders for all domains on each mail server.


Attention! Although this option changes the configuration policy on all servers, synchronization is performed separately from each of the servers (that is, without a single common space)


At the administrator - there will be an opportunity to distribute access between users.
in my case, all of my own and I need complete synchronization (in your case, the solution may be different) on each server you need to create the same sets of domains that need to be synchronized.


2. Kerio Data Directories


Now you need to create the same shared directories, which must be synchronized on each of the servers. Folders, Calendars, Contacts.


Tip - create directories in English, if you create them in Latin, the directory will have a name in some obscure encoding, this is at least inconvenient.


Now you need to find the physical paths of the mail folders on each server.


Common to all CDMY0CDMY domains
Special for each domain CDMY1CDMY


Please note that we will not synchronize the entire catalog, but only the container with the data #msgs - the objects themselves are stored here, all other data for each server must be different.


3. DFS


I’ll also not describe in detail how to configure DFS, I have enough information on this issue.


DFS is a role service in Windows Server that provides the ability to combine public folders located on different servers
MS DFS Document Link


Before configuring DFS - you must stop all mail servers that will participate in data synchronization.


At the end of the setup, you should get the following image for each of the synchronized folders


ITKarma picture


We naturally don’t need to publish replicated folders.


ITKarma picture


After the replication happens (and there’s nothing special to replicate - the folders are empty), mail servers can be started.


Next, you can populate one of the mail servers with data and verify that the data is replicated correctly.


4. Crutch


Description of Reflection


As you can see, after the data began to synchronize (DFS), in the event that you either created something on the first server - something somehow does not appear on the second server, or it appears but somehow not always.


Despair is not worth it, of course, sooner or later it will appear there, but better sooner than later. Because it’s too late in 6-12 hours.


The thing is that as soon as you create something on the first server, the file will appear immediately on the second and subsequent ones thanks to the DFS system, however, if this mail directory has already been read by someone before and it they will request it again, the server will not re-read the #msgs folder and will spit out data from its own index, which may not correspond to our reality for a long time.


Kerio has a mechanism for re-reading the index, but it can work in about six hours, and during these 6 hours the relevance of the task in the calendar may be somewhat lost.
In order to check the operation of synchronization right now, you can delete the file in the corresponding synchronized directory index.fld, after accessing the folder on the mail server again and in the absence of this file, Kerio will re-read the directory and the data will appear. It would seem that here’s the solution, delete the file when the data changes, but this does not work every time, but only the first time, then Kerio somehow loses all interest in index.fld
So she begins to spit out messages incomprehensible to the user - about some kind of index and that he is already doing something there.


There is another option, to create something - at the time of creating a new object, the server suddenly realizes that the name of the file that he wanted to assign is already taken, but there is a snowball and this is a dead end option.


How to be?


If you once again pay attention to the picture we already know.


ITKarma picture


But on a different plane, you can see a very interesting and necessary button for us now - Re-index folders


And really. If you click on this button on a mail server that does not know that it has already changed something in the synchronized #msgs, we will get a stable, fast result. Everything hidden will be revealed.


In the log you can see how long this process takes, in my case with several thousand (15 thousand) records about 3-4 minutes.


All that remains is to figure out how to actually press this button when we need it.


It turns out that Kerio has its own API


Description
Documentation


The function that performs our task looks like this -
CDMY2CDMY


From all of the above, we need to write a script that would monitor the status of the folders of interest and, in the event that something has changed, performs the function we need for us.


I want to say that I wrote several different versions of scripts that perform different checks, I focused on the one that builds all the conclusions based on the number of files.


Implementation of the script


Example CMD script and description


Re-index.bat


@echo off set dir=%~dp0 %dir:~0,2% CD "%~dp0\" md "%CD%\LOG\" md "%CD%\Setup\" ECHO -Start- >> "%CD%\LOG\%Computername%.log" ECHO Start -> %Computername% %Date% %Time% >> "%CD%\LOG\%Computername%.log" SetLocal EnableDelayedExpansion for/f "UseBackQ Delims=" %%A IN ("%CD%\Setup\%Computername%.List") do ( set/a c+=1 set "m!c!=%%A" ) set d=%c% Echo Folder=%c% ECHO Folder=%c% >> "%CD%\LOG\%Computername%.log" ECHO. ECHO. >> "%CD%\LOG\%Computername%.log" :start cls if %c% LSS 1 exit set/a id=1 set R=0 :Find REM PF-Start if "%id%" gtr "%c%" if %R% == 1 Goto Reindex if "%id%" gtr "%c%" timeout 60 && Goto start For/F "tokens=1-3" %%a IN ('Dir "!m%id%!\#msgs\"/-C/S/A:-D') Do Set 2DirSize!id!=!DS!& Set DS=%%c if "2DirSize!id!" == "" set 1DirSize!id!=!2DirSize%id%! echo %id% ECHO !m%id%! echo Count [ !1DirSize%id%! -- !2DirSize%id%! ] if "!1DirSize%id%!" == "!2DirSize%id%!" ECHO Synk REM DEL index.fld if "!1DirSize%id%!" NEQ "!2DirSize%id%!" del/f/q !m%id%!\index.fld && del/f/q !m%id%!\indexlog.fld && del/f/q !m%id%!\search.fld && set R=1 && ECHO RE-index Count && ECHO RE-index Count %Date% %Time% - Delete !m%id%! >> "%CD%\LOG\%Computername%.log" set 1DirSize!id!=!2DirSize%id%! ECHO. ECHO. set/a id+=1 goto Find :Reindex ECHO. >> "%CD%\LOG\%Computername%.log" ECHO --- RE-INDEX - Start - %Date% %Time% --- >> "%CD%\LOG\%Computername%.log" ECHO. >> ----------------------------------- >> "%CD%\LOG\%Computername%.log" call PublicFolders.py timeout 60 goto start exit 

A copy of the script is launched on each mail server (it is possible as a service, Adm rights are not required)


The script reads the CDMY3CDMY file


Where% Computername% is the name of the current server (The directory can contain lists of all servers at once.)


File% Computername%.List - contains the full paths of the synchronized directories, each path is written in a new line, should not contain empty lines.


After the first run, the script performs the indexing procedure, whether it is necessary or not, the script also creates an index of the number of files in each of the synchronized directories.


The task of the script is to count all the files in the specified directory.


At the end of each directory, if at least one directory does not match the current file value, the script deletes the files from the root directory of the synchronized mail directory: CDMY4CDMY and starts the indexing process - shared mail folders.


Information about the execution of tasks is dumped into the LOG directory.


Indexing Process
The indexing process comes down to executing the Kerio API function
Session=callMethod ("Domains.checkPublicFoldersIntegrity ", {}, token)


An example of execution is given in - python
PublicFolders.py


import json import urllib.request import http.cookiejar """ Cookie storage is necessary for session handling """ jar=http.cookiejar.CookieJar() opener=urllib.request.build_opener(urllib.request.HTTPCookieProcessor(jar)) urllib.request.install_opener(opener) """ Hostname or ip address of your Kerio Control instance with protocol, port and credentials """ server="http://127.0.0.1:4040" username="user" password="password" def callMethod(method, params, token=None): """ Remotely calls given method with given params. :param: method string with fully qualified method name :param: params dict with parameters of remotely called method :param: token CSRF token is always required except login method. Use method "Session.login" to obtain this token. """ data={"method": method,"id":1, "jsonrpc":"2.0", "params": params} req=urllib.request.Request(url=server + '/admin/api/jsonrpc/') req.add_header('Content-Type', 'application/json') if (token is not None): req.add_header('X-Token', token) httpResponse=urllib.request.urlopen(req, json.dumps(data).encode()) if (httpResponse.status == 200): body=httpResponse.read().decode() return json.loads(body) session=callMethod("Session.login", {"userName":username, "password":password, "application":{"vendor":"Kerio", "name":"Control Api-Local", "version":"Python"}}) token=session["result"]["token"] print (session) session=callMethod("Domains.checkPublicFoldersIntegrity",{"domainId": "test2.local"}, token) print (session) callMethod("Session.logout",{}, token) 

http://127.0.0.1:4040 can be left as is, however if you need HTTPS python must trust the Kerio certificate.


Also in the file you must specify an account with the rights to perform this function (Adm - public mail folders) of the mail server.


I hope my article will be useful to Kerio Connect administrators.

.

Source