4DBB-Forum: SET DRAG ICON doesn't return Process ID

If you haven't already, please review the documentation about drag and drop in 4D.

Basically, since v11,

• it is no longer necessary to know the process ID of the source/destination,
• it is no longer necessary to use DRAG AND DROP PROPERTIES,
• it is no longer necessary to perform interprocess communication.

in fact, interprocess, process, or even inter-application drag and drop can be coded the same way.

and it is extremely simple compared to the old style.

in the old style, you put a lot of code in the source process, and you had to do a lot of interprocess dance to get it to work.

with the new style, you need only to
APPEND DATA TO PASTEBOARD in the source process,
and GET PASTEBOARD DATA in the target process.
everything you want to tell the target you convey with the pasteboard.

Miyako Keisuke

Subject: Re: Should this work?
Date: Mon, 7 Oct 2013 05:39:04 +0900
From: miyako

Hello,

On Timer and On Outside Call are both form events.

neither are interruptions,
they are like signals that tell the scheduler to generate an event when it next yields a tick to the target process. if multiple timers and outside calls were made during that period, only the last one is processed, the other events are not queued; therefore lost.
for example, if you declare SET TIMER(1),
and SET TIMER(0) 10 seconds later,
that doesn't guarantee 600 events,
it means the scheduler will make the best effort to yield events for the next 10 seconds.
with today's best processor I think that would be about 200 events, so 400 are lost.

another example is if a timer event and outside call event was requested in a single tick, only one or the other is raised at best; if the scheduler is too busy, i.e. other processes are taking time, and more timer/outside calls are generated in the meantime, both events are lost.

on the other hand, EXECUTE ON CLIENT and JavaScript events in the Web Area are queued, which means if you do something like setInterval(function(){}, 1); 1000 events are generated in 1 second, and 4D will take as much time needed to process them all procedurally.

Date: Fri, 9 Dec 2011 15:55:08 +0100
From: Thomas Maul <Thomas.Maul@4d.com>
To: 4D iNug Technical <4d_tech@lists.4D.com>

The Story about Local and Process Sets - which one is faster?

the story goes on :-)

I had the pleasure to discuss this issue with our master of database and the master of user interface (=both Laurents) and want to share the result.

First some terminology:
$Set = Set stored/handled on the client, visible for one process <>Set = Set stored/handled on the client, visible for all process Set = Set stored/handled on the Server, visible for one process

(So maybe the naming of local and interprocess set is not the best and confusing, think more as a kind of namespace...)

Set:
Internally stored as object, imagine a kind of array with one element per "page". A page is 32768 records. If all records are either in the set or not in the set, an array element just contain the info "all in" or "none in". If it has mixed content, it refers to a memory block of 4096 byte which represents a boolean array (32768 true or false). This object is either handled from the server or from the client but can be transmitted from Server to Client or vice versa.
This also means that a set does not needs a big, continous amount of memory but could be spreaded over the memory in 4 kb blocks (easier to fit in heavy fragmented memory on heavy used server).

Knowing that we can understand that:
- CREATE EMPTY SET: for $Set or <>Set transfers nothing over the network, the client can create the set totally on his own. For (Server) Set a small package is sent to the server asking the server to do the work
- CREATE SET: for $Set or <>Set the server creates the set in his memory, then transfers it over the network (and free the memory), the client stores the received object in local memory. For Set a small package is sent to the server asking the server to do the work. Object is kept in server memory.
- USE SET: for $Set or <>Set the Client sends the whole object to the Server. For Set a small package is sent to the server asking the server to do the work
- ADD TO SET/REMOVE FROM SET: for $Set or <>Set the client can do everything locally without contacting the server. For Set a small package is sent to the server asking the server to do the work
- UNION/INTERSECTION/DIFFERENCE:
if all sets have the same namespace: for $Set or <>Set the client can do everything locally without contacting the server. For Set a small package is sent to the server asking the server to do the work if sets have mixed namespace: Depending of target set namespace set's are copied over the
network and then processed from owner of target set.

Knowing this we can follow:
$Set or <>Set helps to reduce the memory need of the server. If you are low on server memory, they can speed up operations.
$Set or <>Set reduces drastically the network access if you do a lot of Add to Set/Remove From Set.
$Set or <>Set increases drastically the network access bandwidth (only a few, but large packages) for every CREATE SET or USE SET operation.

ok, still with me?
Following that, we can say that a Listbox (based on a selection) should never use a Server Set (="set") as selection set, but always $set (remember that "UserSet" is a $set (=stored on Client), we kept the name for compatibility issues). The same is true for Modify Selection, but here we always use "UserSet". Using a Server Set for a listbox means that for every click (=selection modification) a package needs to be send to the server.

Not always true, but as a general idea:
If you have a slow network, such as DSL, use server set's except if you do a many Add/Remove operations.
If you are low on server memory, use $sets
If you use a set only to "remember" a selection, without modifying the set, use a server set (only two small packages sent to the server for Create/Use) Except in very rare conditions, never mix $ and Server set.

So far this is what Thibaud/Josh and myself already wrote, with some more details how sets are handled internally.
What is different to the original post of Thibaud:
"- Local and IP sets reside only on the client. Create set, add to set, remove from set, and set operations don't trigger any request to the server. Obviously, USE SET does (creates the selection)."

I think that was a typo, "create set" should have been "CREATE EMPTY SET". Client can do that totally on his own, while for CREATE SET the client needs to know the selection from the server. I could imagine (but did not tested that) that the client may be clever enough to know that after a REDUCE SELECTion(...;0) the set has to be empty, but for all other cases only the server knows the content of the selection (the client knows only how many records are in the selection, not which ones).

Enjoy the weekend, Thomas, 4D Germany

Date: Tue, 6 Dec 2011 08:26:06 +0100
From: Thomas Maul <Thomas.Maul@4d.com>
To: 4D iNug Technical <4d_tech@lists.4D.com>

Auto Update of Client Problems

The feature was written to run on any computer without special administrator rights. This is the only way to "run everywhere". Of course it also means that it runs only under special conditions.

These conditions are simple: 4D needs to have full read/write/delete rights into the folder, where the previous client is installed. The old client folder is renamed, then a new one is created. So write access inside the folder is not enough, the enclosing folder also needs to be allowed. Necessary rights are not just read/write for existing files, but also to create new files/folders - and to delete existing ones.
That's why the client folder must not be installed into the root folder of the hard disk, but should work one level below.
There is another one: all path's needs to be supported by DOS on Windows and by Terminal (without a shell) on Mac. So issues with extremely long path names or Unicode characters in path name are possible, just to name an example.

The job itself is "open source" (not speaking about license, but about "you can see the source"). This allows you to check or even to modify the way. If a customer installation needs special behavior - you can do that.

The concept is simple:
Inside 4D Volume Desktop/Resources is a file named "upgcInt.bat" (on Windows, on Mac it is a .sh file).

Open it with a text editor and modify as you like. There are many comments inside, so you should be able to follow/understand what is happening, at least if you have basic knowledge of DOS/Unix.

If you want to have a closer look - or if you need to debug "why does it fail", you can modify the comments.
You will see a line such as:
Rem echo "[p0=%~0] [p1=%~1] [p2=%~2] [p3=%~3] [p4=%~4] [p5=%~5] [p6=%~6]" Modify it to become:
echo "[p0=%~0] [p1=%~1] [p2=%~2] [p3=%~3] [p4=%~4] [p5=%~5] [p6=%~6]" >> %APPDATA%\update_Log.txt

rem means it is a comment, so remove it - and >> redirects the output to a file on disk, here in the application data folder (where we know that we have write access)

insert additional lines such as:
:docopy
echo "create new folder for moving the old client %name%" >> %APPDATA%\update_Log.txt md %5

(the line with echo is inserted, the two other's are already in the code. This helps to see where the code just is)

You may want to add a redirect for results, by exampe replace:
move /Y %1 "%~5\"
with
move /Y %1 "%~5\" >> %APPDATA%\update_Log.txt

Save all changes. Copy that file back into the resources folder. Build you client.
Install it.

Now the tricky part:
If the automatic update feature is run, it uses the OLD (this means the previous version) of the upgcInt.bat file. Why? The new one is not unpacked yet, so it must use the old one.
This means if you do any modification, you need to upgrade twice to get it to work. And: you cannot modify that file in a client, you need to do it before building. The built client is "protected", the source is not "open" anymore.

As the installation process is handled from a batch file, you can modify it following your needs.

If you want to install inside the "Program files" folder, you need to have administrator rights, and following the Security concept from Microsoft, this means the installation cannot be silent, somebody with admin rights needs to agree, it cannot be hidden. If you want to have an installation inside protected folders WITHOUT admin rights, you need to use a software deployment concept on system level. Either install during the login phase (the group policy from Active Directory allows to do that, even it is not really nice) or use professional software for this.
There are even Open Source projects - and some can be running on a preinstalled virtual machine, so not so difficult to use, such as: http://www.opsi.org/en/features/software-deployment

Of course such a software needs to be approved from the company first, then it needs to be installed and configured (from somebody with company wide admin rights) - and then you can use it for your deployment.
The automatic update feature of 4D Client does not need and approval, does not need and configuration or installation, but has limits as it does not need admin right.

I know, the perfect solution would be to have a software which can install anywhere and does not need any admin approval, but this kind of software is named "virus" or "Trojan horse" and I don't believe this kind of code fit's your customer requirements :-)

Hope this helps,
Thomas

Date: Mon, 24 Oct 2011 07:54:45 +0200
From: Thomas Maul <Thomas.Maul@4d.com>
To: 4D iNug Technical <4d_tech@lists.4D.com>

Squeezing Performance out of Multiple Processors

Why don't you simply try it?
We have published a movie showing that 24 cores are used - but sometimes people need to see themselves with their own data to believe :-)

To reproduce:
Use a database with a large table or several tables. If you want to test your hard disk, have a larger datafile than cache. If you want to test CPU, have the data file fit into the cache.
Connect several clients. On a 8 core server one fast client is enough.
Write a method which queries random in that/these tables.
Run the queries in a loop, let's say for 10 minutes. The loop just queries, nothing else.

Open task manager on the server. If it is Windows 7 or Windows Server 2008 Enterprise edition also open "Resource Monitor", page on Disk.

Start on the client the method in a new process. The Server will use one CPU/Core with 100% (because the client fires as many queries as possible, as fast as possible).

Start on the client the method in a new process. The Server will use two CPU/Cores with 100% each (because the client fires in two processes as many queries as possible, as fast as possible).

Continue to launch processes, as your cores become busy.

If your CPU's are not 100% busy but you have more processes running as cores/CPU's, check "Resource Monitor". You will see that the waiting queue for the hard disk is > 0.5. The CPU is now waiting for the hard disk, even it is idle. To resolve this, buy a faster disk (SSD) or more RAM.

That's quite easy to test.

You may want to show on the clients a loop counter, so you can see how many queries they do. This is not for benchmarking, as the speed fully depends of your data (indexed or not, kind of data, etc, and the query condition), but it could be interesting as soon you run more queries than cores. Let's say it run's 1000 queries/second per core and you double the number of processes (16 processes for 8 cores), than you will notice that the speed drops to 450-520 queries/second. Add more cores, and the speed goes up again. Scalability...

We showed that during a Summit (2009?) with a tool which allowed to disable Cores online, so we could remove/add cores, showing how speed went down/up.

So try yourself.

Summary: if you have 2 clients and 8 cores, waste of money (Server hardware). If you have 50 clients and 4 cores, waste of money (but now salary expenses). You do not need more cores than working people. If you have 50 peoples doing phone calls and using the computer once per day, you don't need to invest in cores.

Yours Thomas Maul

4D server Monitoring application

http://exchange.nagios.org/directory/Databases/Plugins/4D/ACI-4D/details they are still using it? I thought we discussed that already.

Using this way is the best way to generate network trouble (like disconnection issues) with 4D v11 or newer (while it produced high server load in v2004, but no network disconnection)
Packets like that was the main source of all that network/disconnection trouble we saw with v11.4-11.6.

If you do it once per 5 minutes, you will not notice network issues, because they don't have impact. If you do it once per second, you will, even possible with 11.9/12.2.

When 4D Server receives a TCP Session (opening a session) on 19813 it waits for the correct "hi, I am 4D Remote version xyz" packet. If it does not receives that in a given timeout, it cancels the connection.
If there are too many opened, but not answered, sessions, we have a kind of Denial Of Service Attack.
One of the main changes done after 11.7 was to reduce the timeout, so reducing (normally) the number of open sessions.

So please don't produce network problems by testing if 4D Server is still alive - you have the risk to kill the service just by doing so. Imagine somebody is calling your front desk every 5 seconds to check if the phone line still works...

The correct check is NOT to call the front desk and ask a human if the line works, the correct way is to check the line technically.

So for a network connection, don't use a session based protocol as TCP, but use a non session protocol, here UDP. 4D Server will receive a packet - and if it is still alive it will answer it, done. No session opened.

An example is in the 4D Internet Commands documentation, chapter UDP Commands, Overview

In Unix language (which should be best for most monitoring tools):
You can poll a 4D Server using a single line shell script on OS X (or Linux): printf "%-32b%-32b%-32b" "\0" "4D Server II\0" "\0" | nc -w 1 -u 192.168.93.76 19813
(change 192.168.93.76 with your server ip address) What it does is sending a 96 bytes structure to port 19813 using UDP.
The structure contains three 32 bytes null terminated string fields.
First one is the machine name. You can leave it blank.
Second one is the service name. 4D v11 and v12 use "4D Server II". 4D v6.5 and up to 2004 use "4D Server 6.5".
Third is the database name. You can leave it blank.

In return you'll receive the same structure with all three fields filled by the server.

This technique works on all versions of 4D Server starting at least with 6.5.

This is the only correct way to check if 4D Server is working on port 19813 (main connection port).

Please ask your customer to change the checking, to avoid disconnection problems.

best regards
Thomas
4D Germany

Thomas Maul in 4D_Tech Digest, Vol 26, Issue 32: Our first v11 beta site

Isn't it also true that in Unicode mode on Windows you get the advantage of a new memory manager that can, in some cases, be much faster than before, and that is not tied to Altura?

No, I think this is a misunderstanding.

What's only available in Unicode mode is the new text reference concept.

Let's say you do a:

ARRAY TEXT($mytext;1000)
$mytext{1}:="x"*30000 `(30kbyte)
$mytext{2}:=$mytext{1}

So you have large text variables, which are assigned to other variables. Best example here is the "bubble sort".

In "non Unicode mode" 4D copies 30k memory to another memory location. If that is done in a loop, like for a sort algorithm, there is much memory to move - and to allocate and to manage. That needs time.

In "Unicode mode" 4D works very different (this is a kind of memory manager, but only for text and pictures).
A text is not assigned to a specific variable, but created as object in memory, then the variable get's a reference to the object (and an internal counter is incremented).
If the variable is assigned to another variable (as example above), simply the reference is assigned and the counter is incremented again. No memory is moved/copied ! No memory allocated! Nothing to do = fast.

If the variable is cleared, the counter is decreased. If the counter reaches 0, the memory is released. Done.

All this is fully transparent, a 4D developer don't need to care (or even to know) about that.

Only reason to know is that you do not need to make your own optimization any more. Some developer had created picture arrays and then used pointers to this pictures, trying to minimize the memory usage, to have each picture only once in memory. With v11 you do not need to spend time for that, this is done automatically, simply use the pictures. In variables, in fields, or in forms. They are kept only once in memory.

But to answer "much faster": yes, if Unicode mode is enabled, usually text management is much faster. First because of the reference concept - and because 4D does not need to encode the text for every line of code.
In most cases you do not see a speed improvement, but as more text handling a method is doing, as more you may realize.
Hugest improvement is Text Arrays.
A ARRAY TEXT($mytext;1000000) and then ARRAY TEXT($mytext;0) could need in v2004 or v11 with disabled unicode a noticeable time, up to several seconds. ARRAY STRING is much faster here.
With enabled Unicode this is totally different, it needs no time (could not measure that...) So if you do a lot with large Text Arrays, think about Unicode...

4D works internally always in Unicode mode, all data, all variables, all is stored in UTF-16.
A plugin can decide if it wants to receive text in UTF-16, or MacRoman or Windows ANSI, whatever mode 4D is running. The 4D programming language will receive text converted to MacRoman if the "non Unicode mode" is set. This conversation needs time. Only a little bit, so don't worry to much.
Even in Unicode mode old plugins will work (with reduced character set).

In general I suggest to developers to convert to v11 without touching the Unicode mode. First test your application. If all works fine, start deploying. Then, after some months of "all works fine", start with Unicode. There is no need to do that upfront, except you need it urgently to access new markets or to get a new development deal requiring Unicode.

If you start doing it, "wrap" your character related code with some
If(Unicode_Enabled)
  ` use commands like Convert Text or Send Packet using Blobs or Regex
Else
  ` old way, keep your old code, as it was in v2004 and before, like Mac to Win
End if

Test it. If you think all works, start beta test. If there is an issue (there are always some issues with export/import/interfaces to other applications nobody remembered anymore, but used once per month), simply reset to non unicode, so the customer can continue to work and solve the issue. No need to change code, so quick and safe.
Then ship next version.
As long as you still have the old code, as in example above, you can switch back - and you can compare with the old way, this will be necessary in the first weeks of learning.

The first steps to Unicode is often a little tricky, specially for people thinking in 7bit (or even 8 bit) character sets. That is not really related to 4D, many issues appears if you need to export/import or communicate directly with other applications/systems.

But you can win something if you spend that time:

• More speed
• Text > 32000 chars - as soon you get used to that, you will not want to work without
• Regex
• compatibility with other languages like these funny eastern european characters :-)

Hope this helps
Thomas

keyword boundaries/delimiters

Sun, 6 Dec 2009 16:39:05 +0900

the regex meta character for a boundary is \b.

however, by default, ICU is set match \b not at every boundary but at every space. to fully capitalize on the boundary feature, one must enable the (?w) flag. therefore, to emulate the word breaking algorithm, one can use a regex pattern like this;

While (Match regex("(?sw)(.+?)\\b";$1;$i;$match_positions_al;$match_lengths_al))
  $word:=Substring($1;$match_positions_al{1};$match_lengths_al{1})
  If (Match regex("\\w+";$word;1))
    APPEND TO ARRAY($2->;$word)
  End if
  $i:=$match_positions_al{1}+$match_lengths_al{1}
End while

this will break a sentence ($1) into an array of words ($2->).

http://userguide.icu-project.org/boundaryanalysis

miyako

4.12.2010

in case some problematic code exists in the On Startup method, one can edit the 4DLink file and set
skip_onstartup_method="true"
miyako

finnse für die App "meineSuperduperDB.4dbase":

~/Library/Preferences/4D/Favorites v1(1|2)/(Local|Remote)/meineSuperduperDB.4dLink