Hi Mark,

You are right, in our setup Cache' is able to remove/assign the VIP to whichever node is the primary mirror member. All production traffic goes through ECP, so the mirror's VIP is used to simplify some administrative tasks only (just to recognize primary node quicker).

Virtual router is a standard component of vCloud Director, so called Edge Gateway. It sits between a single external address and internal load balancer implemented as a couple of Linux based VMs, each of which has haproxy for balancing and ucarp for VIP/failover. Load balancer distributes incoming client connections among several ECP application servers (Linux based VMs as well). 

That's how it works, in short. If somebody is interested in more verbose description, I could write more about it.

Regards,
Alex

Murray, thank you for the series of articles.
A couple of questions I have.

1) Documentation (2015.1) states that Rdratiois a Ratio of physical block reads to logical block reads,
while one can see in mgstat log Rdratio values >> 1 (usually 1000 and more). Don't you think that the definition should be reversed?

2) You wrote that:
If you do see high PhyRds on your system there are a couple of strategies you can consider:
...
- Long running read only reports, batch jobs or data extracts can be run on a separate shadow server or asynchronous mirror to minimise impact on interactive users and to offload system resource use such as CPU and IOPS.

I heard this advice many times, but how to return report results back to primary member? (ECP) mounting of remote database that resides on primary member is prohibited on backup member, and vise versa. Or these restrictions does not apply to asynchronous members (never played with them yet)?
 

Just adding my 2c to "4. Tip For UNIX sites":
All necessary unix/linux packages should be installed before the first invocation of 

Do run^pButtons(<any profile>)

otherwise some commands may be missed in ^pButtons("cmds"). I've recently faced it at the server where sysstat wasn't installed: `sar -d` and `sar -u` commands were absent. If you decide to install it later (`sudo yum install sysstat`in my case), ^pButtons("cmds") would not be automatically updated without little help from you: just kill it before calling the run^pButtons().

This is actual at least for pButtons v.1.15c-1.16c and v.5 (which recently occurred on ftp://ftp.intersys.com/pub/performance/), in Caché 2015.1.2.

In Cache 2015+ you need not put a new cache.key in MGR directory manually any more, as SMP>Activate New License page has got an option to browse server's file system. After you have chosen the key file, SMP checks it up and copies to MGR automatically. This option simplifies much the searching of right directory, especially for new users.

Another useful method for cache.key analyzing is: 

do $system.License.DecodeAuth("<AuthorizationKey>")

which displays the capabilities encoded in the <AuthorizationKey> field of a given cache.key file (should be copy-pasted from the file). Argument-less method:

do $system.License.Decode()

does the same for active license key.

Thank you for the article, Mark. I have a couple of questions on your "Option 3: Geographically Dispersed Deployments".

You placed two DR acync nodes at Data Center B. So, the data flow from Primary (at Data Center A) to DR async (at Data Center B) will be doubled. In general, WAN is not too quick, so it may increase latency. More important that mirrored databases at both DR async nodes may have different states (due to different latency), and it would be difficult to decide which of them should be promoted as a new Primary.
And how to catch up the database at the new Secondary, if we can't guarantee that it is in the same state as at the new Primary?

Hi Mark,
May I ask you to clarify a bit.
Am I right thinking that journal transfer is acknowledged only when it runs between failover nodes, so network latency is not a great problem for async communications? Of course, the bigger latency the bigger journal transfer delay, but it would not slow down the primary node operation.
E.g., if we have average 10-20 JrnWrts per second during each busy hour (with spikes up to 300), for right async WAN planning I should take in account that the latency <= 50 ms (1000/20=50) should be sufficient if we can live with DR async node that is usually about several seconds behind the primary?
As I can remind, shadowing allows cascading. Don't you think that cascading can be a good option for deploying of several async nodes at one (long distance) location as well?
Thanks,
Alex
 

Thank you for excellent articles, Murray.

We use slightly different approach for memory planning.

Our app mostly runs as a set of concurrent user sessions, one process per user. It's known that avg memory per process is 10Mb, we multiply it by 3*N_concurrent_users. The 1st multiplier (3) makes a gap for memory spikes. So, the result is a memory we leave for user processes.

We try to leave for Routine Buffer cache as much memory as possible, upto 1Gb.

The Global Buffer memory is usually calculated as a 30% of 3-years-old-database size for given kind of customer. Usually it comes to 24-64Gb global cache for medium to large size hospitals and provides thousands (or dozens of thousands) Rdratio. At whole, we usually get numbers that are close to your 60/40 proportion, while my globuff calculation method is not so presized as yours and I feel that I need a better calculation base for it.

I.e., the "]"-operator returns 1 if the first string operand collates after the second string operand.​

That's true for String type collations only (e.g. Cache String, Cyrillic2 String, etc). For most other collations defined in Cache (traditionally called Numeric collations) which comply the rule "numbers go first" (e.g. Cache Standard, Cyrillic2) one should use "]]" as "collate after" operator. In common, "]]" seems to be the better choice as it uses current collation rather than String one. This small sample code can demonstrate the difference between "]" and "]]": 

num new b,a,i,i0  set b("12345678901234567870")="Num20" set b("12345678901234567874")="NotNum20" set a=1E30,b(+a)=1,b(+a_"11")=11,b(+a_"111")=111,b(+a_"1a")="3a",a=3E30,b(+a)=3,b(+a_"33")=33,b(+a_"333")=333,b(+a_"1a")="4a" set i="",i0="" for  set i=$order(b(i)) quit:i=""  write "(i]]i0)=",i]]i0," (i]i0)=",i]i0," (i>i0)=",i>i0," (i=+i)=",i=+i,?30," " zwrite b(i) set i0=i quit

The result for Cache Standard collation is:  

LEARN>d num^ztest
(i]]i0)=1 (i]i0)=1 (i>i0)=1 (i=+i)=1 b(12345678901234567870)="Num20"
(i]]i0)=1 (i]i0)=1 (i>i0)=0 (i=+i)=0 b("12345678901234567874")="NotNum20"
(i]]i0)=1 (i]i0)=0 (i>i0)=1 (i=+i)=1 b(1000000000000000000000000000000)=1
(i]]i0)=1 (i]i0)=1 (i>i0)=1 (i=+i)=1 b(3000000000000000000000000000000)=3
(i]]i0)=1 (i]i0)=0 (i>i0)=1 (i=+i)=0 b("100000000000000000000000000000011")=11
(i]]i0)=1 (i]i0)=1 (i>i0)=1 (i=+i)=0 b("300000000000000000000000000000033")=33
(i]]i0)=1 (i]i0)=0 (i>i0)=1 (i=+i)=0 b("1000000000000000000000000000000111")=111
(i]]i0)=1 (i]i0)=1 (i>i0)=1 (i=+i)=0 b("3000000000000000000000000000000333")=333
(i]]i0)=1 (i]i0)=0 (i>i0)=0 (i=+i)=0 b("10000000000000000000000000000001a")="3a"
(i]]i0)=1 (i]i0)=1 (i>i0)=1 (i=+i)=0 b("30000000000000000000000000000001a")="4a"

Hello Mark,

As I had no idea which date and time functions can be affected with TZ setting, I'd tested some of them using a codelet:

s top=1000000 f fun="$h","$ztz","$zts","$zh" {s ts=$zh f i=1:1:top { s @("d="_fun) } w fun,?8,$fn($zh-ts/top,"",10),! } d $zf(-1,"echo $TZ")

My testing environment was:

%SYS>w $zv
Cache for UNIX (Red Hat Enterprise Linux for x86-64) 2015.1.2 (Build 607_0_15223) Thu Jul 16 2015 17:33:31 EDT
%SYS>!cat /etc/centos-release
CentOS release 6.6 (Final)

The results w/o TZ:

$h      0.0000023653
$ztz    0.0000009993
$zts    0.0000004639
$zh     0.0000003155

and with TZ:

$h      0.0000011856
$ztz    0.0000001379
$zts    0.0000004690
$zh     0.0000003189
Europe/Moscow

It's easy to set TZ for a Linux/UNIX tty user, but what about an app that is running some flavor of client/server mode? In this case Cache process inherits its environment from a special kind of parent, usually from SuperServer^%SYS.SERVER. 
At the moment I have no idea how to set the environment for SuperServer. I've tried:
1) setting system-wide using /etc/profile.d/*.sh
2) setting system-wide using /etc/environment
3) setting for cacheusr user using his .bash_profile.
Running a sample below, I'm getting a nice picture on /dev/pts and opposite one on |TCP|1972. The results are added in comments. I used CacheActiveX.dll (%Service_Bindings) for client/server connection.

tz(fun) if $zversion["UNIX" {   set f="echo $TZ" open f:("QR"):1 if '$test { write 0 quit ""} use f read tz close f   set f="id" open f:("QR"):1 if '$test { write 0 quit "" } use f read id close f } else {   set tz="", id="" } set fun=$get(fun,"$ztz") ;"$h" set top=1000000 set ts=$zhorolog for i=1:1:top { set @("d="_fun) } set res=$zutil(110)_" "_$principal_" "_id_" tz="_tz_" "_fun_" "_$fnumber($zhorolog-ts/top,"",10) ;d $zf(-1,"echo $TZ")  quit res ; ; tap01.sparm.com |TCP|1972|1311 uid=502(cacheusr) gid=503(cacheusr) groups=503(cacheusr) tz= $ztz 0.0000011239 ; tap01.sparm.com /dev/pts/0 uid=504(alex) gid=503(cacheusr) groups=503(cacheusr),10(wheel),505(alex) tz=Europe/Moscow $ztz 0.0000001212 ;

It seems that the source of the problem is a method of [re]starting Cache.

When it is started from shell using `ccontrol start ` command, SuperServer (as well as its childs) recognizes TZ that's actual system-wide.
But when it is started using a service script `service ca_cache start`, TZ is not recognized. There is nothing special in my script, its start() function is implemented as a wrapper for just the same `ccontrol start ` command as in the first case. It seems that service scripts are started from some special environment where some environment variables are deliberately unset.

I fixed the issue by setting the correct TZ in /etc/environment file and including one line of code into the service script:

start() {
        echo "Starting ca_$prog:"

        [ -f /etc/environment ] && . /etc/environment && export TZ
        ccontrol start $prog quietly
 

Jose,

"RemainAfterExit=yes" setting seems to address the loosing service status issue you mentioned. (Have no RHEL/CentOS 7 at hand to check it).

Hello Mike,

Add to your list several books written in Russian and in German:

• M СУБД  - 2013, by Eugine Karataev
• Von ANS MUMPS zu ISO/M - Fortgeschrittene Programmierung in M  - 1993, by Wolfgang Kirsten
• Einführung in die Programmiersprache MUMPS  - 1989, by Stephan Hesse and Wolfgang Kirsten.

=
Kind regards,
Alex

Dmitry, thanks for sharing this info. Fix a typo: here

/usr/share/bash-completion/completions/ccontrol

# bash completions for InterSystems csession

it should be: /usr/share/bash-completion/completions/csession

Murray, thank you for continuing the series.
A little question I have: does quicker journal response in ECP environment depend on whether Mirroring is used, or not?

Mix SQL with pure MUMPS
To feel mismatching
Impedance.

Exit Caché, Ensemble, DeepSee:
It's summer -
Jump into deep sea!

#CachéLimerick

There was an old man in the vale
Whose DB was as fast as a snail.
After move to Caché
He could jump entrechat
And today he is far from his vale...

w "How robust is your great OS",!
 w "(Each of which has cons and pros),",!
 w "But specific under stress?",!
 w "Want to check it? Just key press:",! r *key set OS=$system.Version.GetOS()
 w:OS="Windows" "Are you really its follower? press <RESET> to do failover!",!
 w:OS="UNIX" "Under *NIX you are OK: your job killed, others remain.",!
 w:OS="VMS" "Not aware of VMS, contribute somebody else!"
 ;for i=1:1 set a(i)=$j("",3*1024*1024) w $j(i,4)

Timur, thank you for the series of articles!
It's clear enough that the purpose of your 1st sample was just to introduce map-reduce ideas, but besides you've illustrated concurrent processing technique available in Cache' which can be used apart of map-reducing. From this point of view, the parallel implementation of word count algorithm could be better balanced if the method MR.Sample.WordCount.AppWorkers::Map() would just count words emitting the result to infraPipe. In this case Reduce() method becomes trivial as all it needs to do - just summarize 4 (= number of book volumes) numbers from the infraPipe.