Extensible Storage Features

Here is a nice hot topic to start off the week.
We already talked about the new Revit 2012 API
EStorage or

extensible storage
functionality and presented example code to

store a map or dictionary
in it.

Here are a few other interesting notes and issues related to this which have cropped up, contributed and pointed out by Steven Mycynek:

  1. Intended use of EStorage.
  2. Handling large amounts of data in EStorage.
  3. EStorage is self-documenting.
  4. EStorage is object-oriented in two ways.
  5. Read/Write permissions.
  6. Modification of EStorage data on an element type.
  7. Handling of ElementId data in EStorage.
  8. Retrieving elements with a specific schema entity.
  9. Checking for a valid entity on an element.
  10. One entity per element.
  11. Schemata remain in memory.

The automatic translation of element ids is one of the absolute highlights of this new technology that I was previously not aware of.

1. Intended Use of EStorage

Question: When should I use EStorage versus the existing technique of storing my data in text form in an XML-based hidden shared parameter?

Answer: EStorage is for storing a lot of complex bits of data that you want to organize into a class-like structure, complete with units, documentation, etc. If you already have an XML file that does all of that already,
and you don’t find using a single hidden shared parameter to be a burden, you might want to go ahead and keep using it. On the other hand, if you didn’t already have an XML schema in place and had a huge variety of data you
didn’t want to convert to a text representation, I’d recommend starting out with EStorage.

2. Handling Large Amounts of Data in EStorage

Question: I am thinking of storing a large amount of data on a number of BIM elements.
What approach would you recommend?

Answer: The intended and recommended use of EStorage does not include storing huge amounts of data in the model. For instance, we have seen issues with developers trying to store very heavy analysis data in hundreds of MB spread across thousands of elements, all loaded at start-up.
Such usage will degrade performance.
If you store large amounts of data across thousands of elements, do not expect an instant load time.

If you wish to store a large amount of data on individual elements, this should not be a problem. For instance, storing something like a .png file on a wall element is no issue at all, whereas it would be an issue to store a dozen .png files on every wall element and extract all of them at once, even if you only need the data for one at a given time.

One of the strengths of EStorage is its handling of arrays of objects or sub-entities and different schemas in general rather than one large segment of data that needs to be read in its entirety and deserialized all at once. I recommend taking advantage of this and only loading what you actually need for a given task. For small loads of data, this might not seem necessary, but when you get into many MB of data, it makes a difference.

3. EStorage is Self-documenting

When you create a schema, you are creating documentation.
Be sure to fill out the documentation strings for each field – they will help you in your development process and others when they use your schema.
What’s more, since you can look up a schema by Guid, if you want to share a document with a schema with someone else, all they need is the Guid to look it up and read your structure and documentation comments.
This might be a good opportunity to either use the SchemaWrapperTools included with the ExtensibleStorageManager SDK sample (or a simpler, similar tool) to print out a schema’s field definitions and documentation strings from a single GUID input.

4. EStorage is Object-oriented in Two Ways

Not only is a given schema entity structured into named fields, but each entity is placed on elements relative to the data itself, as opposed to one large blob that you must read in its entirety to unpack.
This goes along with what the recommendation above about not reading all storage at start-up.
Since you can choose which elements to process, you have the opportunity to only load what you need an automatically have an association between data and a specific element – shared parameters can’t do this.

5. Read/Write Permissions

This is another area that goes beyond shared parameters.
While your schema definition is public, you can restrict who reads and writes schema data based to a specific vendor or a specific application from that vendor.

6. Modification of EStorage Data on an Element Type

Question: When transferring types from one project to another using Transfer Project Standards, it only copies across types that are different. If you change a parameter on a wall type that exists in both projects, then it gives you the
option of copying over “new” types or overwriting existing types. However, if the parameters are unchanged but the schema data is different, it treats the wall types as identical, so does not give the option of copying over the
wall type.

Are element types with different EStorage data attached to them treated as the same or not?

Answer: Any parameter based change, hidden or not, will trigger a new type to be recognized. EStorage, however, is not a parameter-based transaction, so those rules don’t apply.
Therefore, element types with differing EStorage attached to them are still treated as the same in this case.

7. Handling of ElementId Data in EStorage

Question: What happens to element ids stored in EStorage?

Answer: When you store an ElementId using EStorage and that ElementId gets remapped because the element is deleted or updated, e.g. because of a worksharing update, your stored ElementId is also automatically remapped to the new ElementId value. This is one strong advantage for using EStorage over text or raw numbers to store ElementIds – the tracking for element updates is handled automatically, so you can be sure that your ElementIds will remain valid.
If the element is deleted, your ElementId will be set to ElementId.InvalidElementId.

8. Retrieving Elements with a Specific Schema Entity

Question: How can I retrieve all elements that have data from a certain schema attached to them?
Optimally, I think that should be a filtered element collector option.

Answer: Right now, you must do a manual iteration of all elements. There may be a project to add a filter as you describe in the future, but we make no promises about any future features whatsoever.

9. Checking for a Valid Entity on an Element

Question: If I register a schema and then select an element that has no entity for that schema attached to it, I would expect the following call to return null:


Entity ent = e.GetEntity( schema );

It does not.
Instead, it returns a valid entity pointing to a null schema.
To handle this, I expanded my check to this:


if( null == ent || null == ent.Schema ) ...

Answer: Use the Entity.IsValid method to check to see if the entity you received from GetEntity actually has data of a given schema.

10. One Entity per Element

Question: Is only one entity per schema possible per Revit element?

Answer: Yes and no.
There is no way to have more than one entity of a given schema per element via the GetEntity/SetEntity operations.
However, you could always create another schema with more than one sub-entities of a given schema type, including array fields and map fields.

11. Schemata Remain in Memory

Question: Is it true that once a schema has been loaded into Revit memory, it never disappears again until the session ends?

Answer: That is true.
A schema is available per use on the session level, even if a new document is created.
Entities are associated with specific documents.

Many thanks to Steve for all of these tips!


Comments

19 responses to “Extensible Storage Features”

  1. Revit Newbie Avatar
    Revit Newbie

    Hi Jeremy, let me first say that I enjoy reading your blog and it has helped me, as I am just getting started programming in revit.
    I have a (non revit) line class I use and I want to use something like this in my code :
    For Each rfi As RoofFaceInfo In RoofFaceList.GetList
    Dim testface As PlanarFace = rfi.Face
    Dim results As New IntersectionResultArray
    testface.Intersect(line, results)
    however I get an error in the last part (line, results). stating the value cannot be converted from line to revit.db.curve.
    Do you hjave any ideas I could use

  2. Dear Revit Newbie,
    If the line instance that you are supplying is your own class, and not a Revit API class, then obviously it is not derived from Revit.DB.Curve, and thus does not fulfill the expectations of the PlanarFace Intersect method.
    Yes, I have a suggestion: convert your non Revit line data to a Revit Line, e.g. using one of the Autodesk.Revit.Creation.Application.NewLine, NewLineBound or NewLineUnbound methods.
    Cheers, Jeremy.

  3. Dear Jeremy Sir,
    Pls suggest how to set Family Parameter value (Retrieve from extensible storage) in Project Editor.
    Thanks & Regards
    Namit Jain

  4. Dear Namit,
    I do not completely understand your question.
    Are you thinking of reading data from estorage in a family definition and using that to populate the family instance parameter value in the project when it is inserted?
    That sounds like a useful thing to want to do. Please confirm, and I’ll look into it.
    Cheers, Jeremy.

  5. Hello, Jeremy.
    I need your help again:)
    I want to store some data in Family and Family Symbols using Extensible Storage. Then I want to save this family to file and use this family in other projects.
    There are no problem to set some data in the Family and its Symbols:
    ….
    Entity entity = family.GetEntity(schema);
    if (entity == null || !entity.IsValid())
    {
    entity = new Entity(schema);
    }
    entity.Set(field, “Family some value”);
    family.SetEntity(entity);
    ….
    foreach (FamilySymbol symbol in family.Symbols)
    {
    Entity symbolEntity = symbol.GetEntity(schema);
    if (symbolEntity == null || !symbolEntity.IsValid())
    {
    symbolEntity = new Entity(schema);
    }
    symbolEntity.Set(field, string.Format(“FamylySymbol value: {0}”, symbol.Name));
    symbol.SetEntity(entity);
    }

    Next I save family. (Edit family – Save)
    But then I load saved family into other project my data in FamilySymbol is absent.:(
    ….
    Entity entity = family.GetEntity(schema);
    if (entity == null || !entity.IsValid())
    {
    return Result.Failed;
    }
    var str = entity.Get(field);
    //here everithing ok. str = “Family some value”
    ….
    but
    ….
    foreach (FamilySymbol symbol in family.Symbols)
    {
    Entity symbolEntity = symbol.GetEntity(schema);
    //here entity is always not valid.
    }
    ….
    Is there way to keep data in Family file for each FamilySymbols?
    I tried shared parameters, but they also doesn’t save in Family file.
    Best regards, Victor

  6. Dear Victor,
    I would say that your question is demonstrating a basic misunderstanding of families and symbols.
    Think of it like this: a family is a database table. The family parameters are the database table fields, and each type in the family is a record. The family types in the family definition are not represented as individual Revit elements and thus do not really exist.
    When a family is loaded into a project, the family types are converted into family symbols, or rather cause them to be generated. The family symbols exist as individual Revit elements, but only within the project into which the family was loaded.
    If you attach extensible storage data to the family symbols, it likewise only exists within the project into which the family was loaded.
    The family file remains unaffected.
    I hope you understand what I mean and see the error of your ways :-)
    I trust you will find an efficient alternative approach to fulfill your requirements.
    Merry Christmas and Happy New Year!
    Cheers, Jeremy.

  7. Hi Jeremy. Thanks for your response.
    Now I understand how families and symbols works.
    I made same suggestion then I tried get Symbols of Family when document.IsFamilyDocument. In this case Symbols is empty.
    I can see two ways to achieve my requirements:
    1) use BuiltInParameters of FamilySymbols (it saves in files). But I don’t like this approach
    2) Use ExtensibleStorage for Family and save needed Symbols info in it.
    I’ve chosen second way.
    I’ve created two schemes: one for family and one for symbol. In symbol schemes I save needed information for each Symbol.
    For example, http://pastebin.com/M1XhkZg0
    Family schema contains only one MAP field where key is Symbol.Name and value is SymbolEntity: http://pastebin.com/CetyXhh2
    I’m already applied this approach in my solution and it works well. Now when I save family and open it in other project I can retrieve data that I saved in Family.
    Happy New Year!
    Best regards, Victor.

  8. Dear Victor,
    Wow, that was quick!
    Congratulations on solving this!
    I would have chosen the same solution as you did :-)
    Happy New Year!
    Cheers, Jeremy.

  9. Hi, Jeremy.
    Extensible storage is a great feature but I have a problem again using it.
    I have a simple test project. I SetEntity to the Family in this project. Save file. Then I delete schema which belongs to this entity and save project again and close it. But then I reopen this project I get error: “Element XXXXXX became corrupt at some time before this session. To continue with this project, save a recovery file, this element will be deleted to fix the problem.”
    The XXXXXX element – is the element which I SetEntity before.
    But if I close this message project doesn’t loaded and Revit close with error.
    in journal I see next logs:
    http://pastebin.com/APwMp5N8
    So, it is very bad. How can I avoid this behavior? I’m afraid if I install my add-in that use Extensible storage and designers will get this error and cannot open file they kill me:)
    May be exists some utility from Autodesk that can recover corrupted file?
    Is this a bug?
    Regards, Victor

  10. Dear Victor,
    You really are diving in deep with the extensible storage.
    Congratulations again on discovering this issue. Your testing is good and important!
    I hope very strongly that you are running on the initial release of Revit 2012, or only the first update release. Some issues with extensible storage were fixed in Revit 2012 Update Release 2:
    http://thebuildingcoder.typepad.com/blog/2011/10/product-and-add-in-wizard-updates.html#2
    You should ensure that you and your customers are running on that version or higher to avoid the issue.
    Happy New Year!
    Cheers, Jeremy.

  11. Good afternoon, Jeremy.
    I don’t know good it or bad but I use last version of Revit Architecture. I’ve read Enhancements list of update 1 and 2 and I hope that there are no errors in ExStorage after update but I was wrong:(
    I’ve already got this issue:(
    P.S. I really doesn’t remember when I installed Update 2. May be installation was not correct. What do you think if I’ll send you this file and you’ll test it in your computer?
    Thanks, Victor

  12. Dear Victor,
    Sounds pretty bad to me if there is any such issue still in there.
    If you really are using the most recent version and your ADN membership has come through by now, then please submit an ADN DevHelp Online case for it. I am on holiday tight now.
    Thank you!
    Cheers, Jeremy.

  13. Hello, Jeremy.
    You write about strong advantage of EStorage when store ElementId there because entity ElementId value mapped with real Element. I fully agree with you.
    But now I’ve found if I rollback transaction ElementId doesn’t remapped to entity.
    That I mean.
    I have a schema with ElementId field. I store Level.Id there but it is not important in this case. Then I create an entity and set it to some element. Next I delete level. So, if get my entity and extract field value, I’ll get ElementId.InvalidElementId because I’ve deleted element before. But if I rollback transaction I also get ElementId.InvalidElementId field value. So the ElementId does remapped with entity.
    How do you think, is it a bug?
    Best regards, Victor.

  14. Dear Victor,
    Thank you very much for noticing and reporting this.
    It definitely sounds to me like something the development team should have a closer look at.
    Could you please provide a reproducible case and submit an ADN issue for further investigation of this?
    Thant would be a great help!
    Thank you!
    Cheers, Jeremy.

  15. Hello, Jeremy.
    Yes, sure. Case #07060034.
    In my opinion Extensible storage use DMU for updating ElementId in entities that doesn’t support transaction rollback. But maybe I was wrong.
    Best regards, Victor.

  16. sangsen Avatar
    sangsen

    Hi..Jeremy
    I am creating “Group” through API and setting some entities to the grouptype. When I save the group locally and reuse in some other document(using Load as Group option) new group instance doesn’t contain any schema information. But i need to retrive that entities?
    please advise me..How can i acheive this?
    Thanks

  17. Dear Sangsen,
    Yes, indeed, schema information is not automatically copied by Revit. That is left up to the add-in to handle as it wishes, e.g. using a dynamic model updater DMU.
    Cheers, Jeremy.

  18. +1 for a filter for retrieving elements with a specific schema entity. It would be very useful. It’s my second project where this kind of filter could have helped me.

  19. Dear Maxence,
    I get the gist of what you mean, i.e. the importance of being able to retrieve elements with extensible data for a specific schema. You will be happy to hear that we are working on that.
    I don’t know exactly what you mean by “+1 for …”
    Does that refer to ‘top priority’, or ‘one more request for’, or what?
    Cheers, Jeremy.

Leave a Reply

Discover more from Autodesk Developer Blog

Subscribe now to keep reading and get access to the full archive.

Continue reading