Bryan

I'll take the opportunity to explain a little more about the benefits I see in extracting the information that's encoded in a 4D structure file. There are some key benefits, and then there are some peripheral ones; I believe most 4D developers would agree about the keys, but different developers will find some peripheral ones more compelling than others.

I really wish it wasn't necessary, but as I see it there are several key benefits to decoding the 4D structure file.

1. Security (as in, Peace of Mind, as well as in Safety)

This is the biggest benefit, I believe. Whilst there's nothing inherently unstable about a 4D structure file, we have to acknowledge that without 4D itself the structure file is completely useless. In fact, without the correct _version_ of 4D, it's useless. What 4D SA has failed IMO to take into account is that inside that structure file is _my_ intellectual property [IP], encoded away so that no-one can get at it (including me, even though it belongs to me). Everybody's business depends on their IP, but if by some mishap or corruption the structure file stops working, they have a really big problem. Backups are good as far as they go, but with 4D your backup is a whole version all at once. This inevitably means that you always have Work In Progress [WIP] that isn't backed up, and if that dies you've then got a potentially large task (depending on how fast you work and when your last working backup was) to get back to where you were.

Extracting this encoded information to plain text does things that I believe are really necessary for a business owner's peace of mind, among them: a. It allows the business's IP to be stored outside of 4D in human-readable format. One is no longer dependent on being able to load the structure file with a copy of 4D to be able to see the IP that's in it. I'm sure you've a good idea how valuable this would be to the business owner, given that you do lots of work with legacy systems whose 4D development environment no longer works on the latest and greatest MacOS or Windows. Storing the IP externally gets rid of this problem. b. It also (by extension) allows the IP to be re-constructed in a different format if required. I think 4D SA may view this as a problem, but actually everyone I know views the locked-in nature of a 4D program as the problem instead. Unlocking it by turning it all into plain text somehow relaxes people and helps them believe that they are not being shackled by a system (whether they are in fact or not). But when the system encodes everything they do and locks it away so they can't get at it except by using the system, they already know that they are shackled, and they move on to another system that doesn't do this.

2. Conformity.

Everyone who uses 4D acknowledges that 4D is "different". Whether this is good or bad is debatable, but the point is that you can't just transplant a developer who's used to working in one of the mainstream IDE's and stick him/her in 4D without also needing to do a fair bit of training. The Explorer view in 4D is nice, but it's not as good as the similar thing in Visual Studio or Eclipse or XCode.

The reverse is also true, although it's worse. From my own point of view being skilled in 4D has meant that I could always find a job doing 4D. But it has also meant that I can't find jobs doing C++ or C# without also getting to know the development environments that make those languages work. This is extra overhead for the developer, made no easier by the fact that 4D is such a non-conformist system. Developing in those languages is not just about learning the language - it's also about becoming familiar with the vagaries of setup and the IDE, which for me at least is a non-trivial task.

These two issues are minor points compared to this one: the fact that 4D is "different" (i.e. uses its own IDE) means that it consequently has very low visibility outside of France. Here in Australia (pop. ~20million) there are probably less than 100 developers who actively earn a living doing 4D. The rest (who knows how many thousands) use other languages. No-one outside of my company that I've ever talked to has even heard of 4D. Everyone has heard of C#, Java, C++, JavaScript, HTML and so on - because you work on them all in the same way, and the barrier to entry is thus very low.

The other "difference" about 4D (and this is not strictly on-topic, but I'll say it anyway) is that it's basically a closed system, and it doesn't play nicely with other applications on either of its supported operating systems. People will object to my saying that, but it's true anyway: show me how you can use a 4D program as a COM object in Windows (or integrate with a .NET assembly, ha!), or send AppleScript commands to it on MacOS. There is no direct support for this in the 4D Runtime. We do communications from .NET applications to 4D applications and back all the time, and we've had to resort to polling known locations for messages in text files. Web Services are obviously available but support for all the twiddly bits is sadly lacking in 4D, so there again you just end up using the lowest common denominator.

All this closed-ness really works against 4D when it's being evaluated against the perception of openness in its competitors. For many people, they see the closed functionality, and may be prepared to live with it if the development system is good enough. Then they see the closed development system as well and they just up sticks and go elsewhere, regardless of how good or how easy the system is to use. I know that if I were learning 4D now that would be a major concern for me.

I love the new code editor in 4D v12. I think it's a massive improvement over the previous incarnations of the same thing, and kudos to the people who put it together. But I have to acknowledge that Visual Studio, Eclipse and XCode all have customisable text editors (of which the style of the new 4D text editor is a copy) linked to an IDE, and the process of making them work with a new language is a well-travelled path. There's no need to invest millions in a new text editor for your custom IDE if you just make your language and forms compatible with the mainstream IDEs that are already hugely popular. Then, the upside to conformity is that you almost automatically get publicity, because people can run your language in the environment that they already have.

Imagine for a moment that 4D SA were to create a command-line version of the 4D Compiler, that could take XML form definitions and other structural information, together with plain-text methods, and turn it into a 4D program. Perhaps even publish the 4D Runtime as a DLL/Shared library with documented entry points, and offer a plugin to various IDE's that makes code to create and drive 4D forms. Then imagine that you could use Visual Studio or Eclipse or XCode to develop with. The act of conforming to what most of the world already does is a tremendously valuable service to the users of your system because it means that any generic tools (source code management anyone?) that already work with the mainstream systems will also work with 4D. And 4D would not have to lift a finger to make it possible.

[People may argue that 4D has already done this with the 4D Open for C library. But there the issue was that you had to work in C, and call 4D via the library. People work in 4D because they don't want to work in C, so making them work with 4D only via C is not really going to go down all that well. Plus, I'm pretty certain that it only exposed a limited set of functions rather than the full 4D API, so there would always have been some important things that were impossible.]

Conformity automatically means a degree of openness, and that openness in turn creates opportunities. Closed-ness denies them, and it's to 4D's detriment that it has remained closed for so long.

3. Source code management (SCM).

As everyone is aware by now, SCM is vital for programming. There are hundreds of tools that support SCM, precisely none of which work with 4D out of the box. This is a major shortcoming that I would prefer to see addressed yesterday, but in the absence of action from 4D SA, about a dozen of us have made the effort ourselves. But it's ridiculously hard to get 4D to work well with SCM systems. The non-conformity I mentioned in (2) really hurts us here, and it means that for any coding effort where SCM is a requirement, 4D will be rejected almost immediately without going into any of its nice features.

I believe SCM is what has made the Open Source movement as powerful and effective as it is. It's not too much to say that without CVS, there would have been no Open Source. Now that we have Git and Mercurial, things just get easier, and it's high time that 4D joined in. This as I've already mentioned elsewhere has been one of my goals for Reflector, and I'm pleased to say that the next version will extend the SCM functionality (such as it is) to forms (in XML) as well as methods. But that is pretty shallow compared to the full SCM functionality that you get when programming in C# or Java, because _everything_ in those systems is just text. The Visual Studio IDE when it's working with C# is really just a code generator. There's no reason why the 4D IDE couldn't be the same.

I'm of the opinion that branching doesn't make sense in the 4D environment as it now stands. Yet any developer would easily see that branching is fundamental to being able to deliver a working product whose expanding features list doesn't come at the expense of fixing bugs. In other systems, when you create version 2 of your SuperProduct, you simply make a branch in the source tree. You continue to maintain the original code for as long as appropriate (trunk), and when you're ready to include the new features in the main product you merge the branch into the trunk. This is of course a gross over-simplification, but you get the idea. Incidentally, I would be using Mercurial or Git for this rather than CVS or Subversion, because they are much superior at the essential process of merging changes.

The equivalent process in 4D involves copying whole structure files to separate projects, maintaining them separately, and _manually_ merging changes. This is unbelievably error-prone and risky, made doubly so by the component architecture in 4D v11+. It's so inconvenient that most people simply don't do it, which in turn means that 4D programs are almost by definition really hard to maintain and enhance at the same time.

Those then are the major benefits that I see from decoding the 4D structure file. The following points are more minor, but still important to me. Other developers will no doubt have their own list or want to rearrange mine into a different order.

4. Analysis.

Programming (in my case at least) is a continuous process of refining things that nearly work into things that really work. This process is not easy in 4D, not least because there are no tools to help. Although SanityCheck made a noble effort to codify some analysis of your 4D structure file and offer helpful hints about how to improve it, it was very much a voice crying in the wilderness, and has disappeared altogether with the advent of 4D v11 and v12 (I know there's a beta program somewhere, but v12 is out already). I don't mean that nastily: binary formats are really hard to pick apart, and it's amazing that Paul Carnine got as far as he did. I have not even approached his standards in Reflector, and I've tried pretty hard, so it's no surprise that the latest SanityCheck is taking a while. If the structure file were plain text, however, I'd guess from my experience that Paul's task would be about 30% as large as it is now.

One kind of analysis that's often helpful is the "lint" (static) kind that checks your program to make sure you didn't do dumb stuff like leave XML references dangling or dereference pointers to deallocated objects. Static analysis can also tell you other things about your program. For instance, how many variables are never used, or whether you've used a process variable outside the scope of its module. It could alert you to the fact that you're assigning values to an interprocess variable without semaphore protection. It could help rearrange the logic of your deeply-nested If Else End if statements to make them easier to read, refactor, and maintain. It could offer to take a block of code and make a function out of it. It could look at the method you're editing and tell you whether you've declared all your variables, or whether the process variable you're relying on is modified by something else. It could produce a nice graph of who-calls-who in your method call chain. There are no tools that do this for 4D; there are hundreds that do it for C#, Java, and the rest. Decoding the structure file is the first step to building a tool to do this kind of analysis.

Still other analysis is of the performance (dynamic) kind. Various tools exist in other development environments that alter the code generated by the compiler to enable the dynamic tracing of the program as it runs, collecting stats about where it spends its time. The only tool we have for this in 4D is the debug log, for which Josh did a nice tech note and presented a small program to analyse its findings. But that was post-facto; the latest stuff in XCode and Visual Studio allows you to do this analysis in real time. Imagine being able to insert code into every method that would automatically track its parameters and execution iterations/time, presenting results in real time. Decoding the structure file makes this possible for legacy programs, not just those (like Reflector) where it's been designed in from the start.

5. Automatic documentation.

Lots of programmers are great at creating code but hopeless at documenting it. Decoding the structure file will enable us to set up templates that can then automatically produce documentation at different levels: for example, Executive Summary, Program Manager, and Maintenance Programmer. I would have a hard time doing all that by hand.

6. Diagramming.

This is a different kind of documentation, also automatically generated.

People have all sorts of different learning styles. Some of us learn by getting straight into details, then building up to the big picture. Others like it the other way around. Others prefer some combination. One tool to assist with this is the diagram. This can be as simple as a method-call-chain diagram, or as complex as a full Abstract Syntax Tree of your 5000-line method that does all the work. Decoding the structure file will enable this extra learning tool to be created. [I have plans of this nature for Reflector, but don't hold your breath!]

7. Bug tracking.

This is closely aligned to SCM. There are lots of bug-tracking systems available. I use Axosoft OnTime, and it has a neat feature where the bug tracker integrates with your SCM system, so that you can point to a bug, discover where in the source code it came from, and show by means of the revised source code that you've fixed it. You can also track features this way. I hope one day to be able to do this in 4D. I've been using 4D almost continuously for over 20 years now, and I'm still not very good at estimating how long a given thing is going to take me to do. This is mostly my fault, for not keeping an accurate diary of my programming time; but imagine if there were a tool (Axosoft is but one of many) that could track your 4D source _per feature_, time how long it takes to create, and track how many bugs came out of it. I would use that every day, hoping thereby to improve both my coding practices and my estimating accuracy.

8. Code sharing.

Again, closely linked to SCM. As I've already said, Open Source might not have happened if there had been no CVS. Nowadays, with my code in a Mercurial repository, I can create related applications by cloning that repository, making changes, and letting Mercurial worry about how to re-integrate any updates to shared code back to the original repository. Try doing that in 4D!

I have a number of small 4D freebies on my web site (http://www.expert4d.com/freestuff). To make them available I have to create a whole 4D database (often in 2 or 3 versions of 4D) and upload the thing to the site. With Mercurial repositories, I could just share the repository and advertise that. Your Mercurial could then track any updates, you could contribute to the shared code, and so on. There is the beginnings of a code community. This is ridiculously hard to do in 4D at present. Even if some kind soul were to publish a 4D Server for the community to contribute to, there's no way to police the code that gets added; no way even to make sure it all compiles before it's accepted to the library.

9. Metaprogramming.

Reflector is a read-only tool. I do not intend to change that policy any time soon, because writing 4D resources reliably is something that only 4D can do, and IMO something that only 4D _should_ do. Reflector will only make changes to 4D's resources by calling a published API function. Since there are none at present, and none on the horizon, Reflector will stay a read-only tool for a while longer.

But if the code and forms were pure text - why, any tool can write text. Metaprogramming here we come!

I could go on... but I'll stop there for now. For me this is a compelling enough list that even if no-one buys Reflector I will continue to enhance it for my own use. I'm pleased that you see the potential for this kind of tool in 4D itself. The principal reason why I wrote it was to put into 4D some functionality that I as a developer find very useful; essentially, I built it for myself. If others find it works for them, that's a bonus.

I will continue to put into Reflector all the things I'd personally like to see in the native IDE (feature requests welcome!); many of which are already available in other IDE's, if only 4D were to become open enough to make use of them.

FWIW, I've tried to write Reflector in such a way that if by some miracle 4D SA opened the structure file format up and made it all plain text (even one huge XML document would be nice) Reflector could be adjusted to work without needing a total rewrite.

I live in hope.

:)

David Dancy Sydney, Australia