Skip to content

SchemaDefinition.Create picks up internal fields instead of just public only - affects F# #6209

Closed as not planned

Description

@fwaris

This used to work before but now I cannot use CreateFromEnumerable in F# now.

In F#, we define mutable classes by annotating F# 'record' types with CLIMutable:

[<CLIMutable>]
type D = 
   { 
         Data :  float[]
   }

F# compiler generates IL that looks as follows:

public class D  [ interface list  ] 
{
        internal Data@ = Double[];

  	public Double[] Data
	{
		get
		{
			return Data@;
		}
		set
		{
			Data@ = value;
		}
	}

     [ more methods including a default constructor ]
}

The SchemaDefinition.Create method picks up both "Data@" and "Data" as fields required by the schema. It should only pickup the public fields.

Please add unit tests for F# as well so these issues are caught earlier.

@dsyme

Activity

  1. ghost added
    untriagedNew issue has not been triaged
    on May 30, 2022
  2. fwaris commented on May 30, 2022

    @fwaris
    Author

    I just found out that one workaround for the above issue is to set "ignoreMissingColumns=true" when calling CreateEnumerable(...)

  3. michaelgsharp commented on Jun 13, 2022

    @michaelgsharp
    Contributor

    Glad you found a workaround, but yeah that isn't the behavior we want I don't think.

    When did you notice this change? We haven't made a recent changes to SchemaDefinition.Create that I am aware of. I wonder if it has to do with a .NET version change? What version are you using now? Could you also try an older version?

  4. fwaris commented on Jun 14, 2022

    @fwaris
    Author
  5. michaelgsharp commented on Jul 11, 2022

    @michaelgsharp
    Contributor

    Since this is a change by the F# tooling there is nothing that we can do about it unfortunately. Closing this for now, but if things change in the future please create a new issue and we will take another look.

  6. ghost removed
    untriagedNew issue has not been triaged
    on Jul 11, 2022
  7. dsyme commented on Jul 11, 2022

    @dsyme
    Contributor

    @michaelgsharp Just to check....

    There are some situations where F# needs to emit public fields - notably when emitting code into multiple assemblies in F# Interactive.

    My understanding is that serialization of fields should ignore

    • internal fields
    • private fields
    • public fields marked "CompilerGenerated".

    This is what System.Json and Newtonsoft.Json both do. Does SchemaDefinition do that? It would be great if it does.

    See also dotnet/fsharp#13494 which is related

    @fwaris Do you know if you hit this issue when using F# Interactive?

    Thanks
    Don

  8. michaelgsharp commented on Jul 13, 2022

    @michaelgsharp
    Contributor

    @dsyme let me reopen the issue for now while I look into that. I am not actually sure off of the top of my head.

  9. ghost added
    untriagedNew issue has not been triaged
    on Jul 13, 2022
  10. added this to the ML.NET Future milestone on Mar 29, 2023
  11. ghost removed
    untriagedNew issue has not been triaged
    on Mar 29, 2023
  12. Szer commented on Jul 20, 2024

    @Szer

    Just in case, it is still an issue as all ML.NET guides advice to use records and they are breaking in F# interactive environment.
    One have to know about hacky workaround passing --multiemit- to FSI which is not a common knowledge

  13. fwaris commented on Jul 20, 2024

    @fwaris
    Author

    Check out MLUtils that has some utility code to help use ML.Net from F#. You can use cleanSchema to remove '@' fields from the default schema

    #r "nuget: MLUtils"
    open MLUtils
    type NRow  = { Field1:int; ...}
    let xs : NRow list = []
    let dv = ctx.Data.LoadFromEnumerable(xs,Schema.cleanSchema typeof<NRow>)

    Going the other way - from IDataView to F# record - ignoreMissingColumns flag works.

     ctx.Data.CreateEnumerable<NRow>(dv,false,ignoreMissingColumns=true)
  14. fwaris commented on Jul 20, 2024

    @fwaris
    Author

    @michaelgsharp Just to check....

    There are some situations where F# needs to emit public fields - notably when emitting code into multiple assemblies in F# Interactive.

    My understanding is that serialization of fields should ignore

    • internal fields
    • private fields
    • public fields marked "CompilerGenerated".

    This is what System.Json and Newtonsoft.Json both do. Does SchemaDefinition do that? It would be great if it does.

    See also dotnet/fsharp#13494 which is related

    @fwaris Do you know if you hit this issue when using F# Interactive?

    Thanks Don

    @dsyme, sorry, did not see this until now.
    There is a workaround. See the last few messages in this conversation for some notes on the MLUtils package.

  15. added
    Priority:2Work that is important, but not critical for the release
    area-CoreCore ML.NET APIs and shared functionality
    and removed on Sep 8, 2026
  16. rosebyte commented on Oct 23, 2026

    @rosebyte
    Member

    Triage: the reported issue has a workaround and we don't plan to extend F# support at the moment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    F#Support of F# languagePriority:2Work that is important, but not critical for the releasearea-CoreCore ML.NET APIs and shared functionalitygreendevelopmentneeds-further-triageNeeds deeper review after initial triage

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions