high-performance schema for science & engineering

It’s advanced and universal?

Easy to use and FREE??

YES!!!

Wouldn’t it be great to be able to access data easily with any of your favorite languages?

Build advanced apps and workflows?

XCOMPUTE utilizes a new strategy (originally developed by Google) to express complex data between computers / sessions as protocol buffers.

When you save or load to disc or transmit something over a network, the associative data structures present in your computer’s RAM must be flattened (aka serialized), buffered, and eventually reconstructed (aka deserialized) so that they can be transmit in linear fashion across a wire or into a storage device and back again.

There are many ways to do this, but most are not suitable to big data.

We’ve elected to use a special protoc compiler to auto-generate compatible interfaces that provides native access across many languages. They’re essentially feather-weight code headers or libraries that allow you to tie into xcompute.

They also sport speeds approaching the theoretical limits of the attached devices and channels (PCIe, etc).

Messages™ by Xplicit Computing provides standard support for:

While xcompute-server remains a proprietary centerpiece of the XC ecosystem, we’re excited to announce our plan to release our other official Apps, free & open!

This way, everyday users do not have to worry about subscription to xcompute-client. It makes collaboration that much easier.

Hosts maintain their xcompute-server subscriptions and now can invite friends and colleagues freely, and share results as they please with said Apps.

You own and control your data, while Xplicit continues to focus on providing high-quality, unified technologies.

For a technical overview, please read this below excerpt from the README provided with the Messages™ bundle:

XCOMPUTE MESSAGES — READ ME Creative Commons License XC MESSAGES 2019
UNIVERSAL HIGH-PERFORMANCE NUMERIC SCHEMA / FORMAT
for complex systems, FEA, CFD, EDA, and computational geometry

A. INTRODUCTION

These proto files provide direct access to xcompute messages (file and wire) by generating accessor functions for your favorite languages. This empowers xcompute users and non-users to be able to directly manipulate and access the setup and data to/from xcompute — in a high-performance universal way — suitable for integration with other applications and scripts. Theses four files are central to the xc ecosystem (e.g. both open and proprietary apps), organized as follows:

  • system.proto – domain-specific parameters, associations, references
  • vector.proto – arrays and linear attributes that benefit from packed arena allocation
  • geometry.proto – topological description of elements and regions for a specific domain
  • meta.proto – user profile and meta-data for a specific system

  • This protocol buffer format can deliever high-performance, elegant, and flexible numerical messaging across many domains of science and engineering. (e.g. single- and double-precision floating point data, etc)

    Benefits:
  • universal formats for storage and transmission between demanding applications
  • cross-platform accessors and serialization utilities
  • flexible associative and vectorized data patterns
  • object-oriented modularity with reverse and forward compatibility
  • supports multi-threaded I/O within vectors and across files

  • Limitations:
  • maximum individual file size is 2GB, limiting individal systems to ~256M nodes (limited by Google’s 32-bit memory layout)

  • Large systems should be decomposed into several smaller systems if possible — for many reasons. It’s more efficient and accurate to specialize the physics, mediating across regions where required. Try to not solve extra DOF’s unncessarily by making one huge domain that solves everything. Memory requirements vary across methods, but is generally limited by your compute device memory…not the storage format or SSD. It is up to each workgroup to determine what is an appropriate resolution for each study. A top-down systems approach is the best way to resolve from low to high fidelity and maintain accountability across the team…

    B. USING XCOMPUTE BINDINGS FOR YOUR PROJECT

    Auto-generated bindings are provided for the following langauages: C++, Obj-C, C#, Python, Java, Javascript, Ruby, and Go. In your language environment, import the relevant files as headers or libraries. Statically-compiled languages such as C++, Obj-C, and C# may require linking to static library libxcmessages.a .

    In your environment, various classes should become available. In C++ they can be found under the namespace Messages:: . Refer to the *.proto definitions for how each attribute is defined, knowing that your access pattern is built from these assignments directly. You can access this associative database using getters and setters…

    In C++, the pattern for accessing primitives (bool, int32, int64, float, double, string) looks like:
    msg.set_something(value)
    auto some_value = msg.something();

    Repeated fields (vectors, etc) can be accessed somewhat normally. Range-based iteration:
    for (auto entry : msg.vector() )
    something = entry;


    Or alternatively for parallel iteration:
    auto N = msg.vector_size();
    something.resize(N);
    #pragma omp parallel for
    for (auto n=0; n'<'N; n++)
    something[n] = msg.vector(n);


    More complex data structures may require mutable getters:
    auto N = other.vector_size();
    //get a reference to mutable object
    auto& vec = *msg.mutable_vector();
    vec.resize(N);
    #pragma omp parallel for
    for (auto n=0; n<'N'; n++)
    vec[n] = other.vector(n);


    Please refer to the Proto3 Tutorials for typical programming patterns.

    C. BUILDING YOUR OWN BINDINGS

    If you’re an advanced application programmer, you may wish to build upon our bindings to customize against your own projects. This is encouraged as long as existing definitions are not altered. Use a Google Protobuf 3 Compiler to generate your new bindings. A protoc 3 compiler may be readily available in a package manager or installed from online sources or binaries. Proto2 will not work, must be Proto3+.

    After protoc is installed, make a directory for each language and run the compiler from shell or script:
    > mkdir -p cpp python java javascript ruby objc csharp go
    > protoc --cpp_out=cpp --python_out=python --java_out=java --js_out=javascript --ruby_out=ruby --objc_out=objc --csharp_out=csharp vector.proto system.proto spatial.proto meta.proto

    the π theorem

    Yes, it’s π day (3.14) and so I am obliged to write a short something on that topic.

    Most people are celebrating the free or $3.14 pizza today. My mother is a little ticked that I didn’t tell her until the evening.

    Academic types are geeking out over the proximity of the base-10 decimal number to today’s Gregorian date. Some might even throw a party at 1:59 local time.

    I’m more excited about Buckingham-π

    Yes, that is lovely. It’d take that over more math any day. BTW, do you think they serve Buckingham-Pi at Buckingham-Palace ? I’ll take two stones for 3.14 pounds please. [source]

    Unfortunately, no. It is a math thing, again! It has awesome science and engineering dimensional-analysis applications (just using units!):

    π-groups are sets comprised of physical variables (with units) raised to some powers so that the group product is devoid of dimensions. (Does not depend on length, time, mass, temperature, etc.)

    This is used to construct an invariant vector space of all physically-consistent variable permutations within the system — hinting at possible underlying relationships or mechanisms.

    Buckingham-π is recommended in multi-dimensional regression studies on instrumented data where there isn’t a clear relationship between measurable physical variables. These groupings imply similar solution space if relative relationships are enforced (asserting analogous conditions). This allows us to do test miniature bridges and wings in the lab, and infer fundamental relationships between dimensional things like like E=mc^2 .

    Shamelessly stolen from MIT notes; the source for everything you don’t know.

    We do this all the time in our heads but less rigorously. We know not to add length and time, but can divide them to make the units of speed. We know you can only add energy to energy, etc. Energy divided by distance has units of force — one of innumerable dimensional truisms.

    As an example, the aspect ratio of your computer screen is a π-group, as width divided by height is dimensionless. n=2 variables and k=1 dimension type (length), so j = 2-1 = 1 possible π-group.

    Non-dimensional parameters such as Reynolds number comprise the π-group for density, velocity, length, and viscosity. We expect self-similar relationships (and system behavior) if the Re is held constant :

    In terms of SI (kg-m-s) units, this equates to dimensional nullity:

    The above has n=4 variables and k=3 dimensions, so j = (n-k) = 1 means this also is the only possible combination for this group. If we had more variables, we would have more possible π-groups and exponential coefficients to determine.

    While integer exponents are expected on dimensional variables, non-dimensional π-groups are technically allowed to have irrational (non-integer) exponents.

    Since 1 to any power is itself, π-groups are invariant under power — the constant exponent relating weightings between multiple π-groups is constant for a specific configuration; coefficients are evaluated using an experiment test matrix for each independent variable.

    Usually, integer exponents can be defined for all by multiplying each exponent by the lowest common exponent as they are often easily divided. In highly-empirical regressions, odd irregular decimals such as 0.37 are not uncommon as they are not factorable.

    Read more about Buckingham-π on Wikipedia